BasedHardware/omi

Features

• Cross-Session Conversation Memories - Persists information and interaction history across multiple user sessions to provide long-term AI memory.
• Conversation Memory Retrieval - Uses vector embeddings to store and retrieve a lifelong history of user interactions for context-aware AI responses.
• Fact Extraction Pipelines - Processes unstructured audio transcripts to extract and store confirmed facts for long-term AI memory.
• Knowledge Base Retrieval - Provides a system for retrieving information from the total historical record of encountered data for context-aware responses.
• Agent Tool Execution - Enables AI agents to autonomously select and invoke external functions to provide comprehensive answers.
• Agentic Tool-Use Frameworks - Provides a plugin-based framework that allows LLMs to interact with external tools and knowledge bases.
• AI Agent Tool Integrations - Provides a plugin framework that allows LLMs to autonomously invoke custom functions and interact with external services.
• Plugin-Based - Provides a plugin-based framework allowing AI assistants to autonomously invoke custom functions and interact with external services.
• Audio Speaker Embeddings - Computes speaker embeddings from audio to enable precise identification and similarity matching of different speakers.
• Audio Transcription - Converts live audio into text using a provider-agnostic system with interchangeable transcription services.
• Real-Time Transcription - Converts streaming audio to text via WebSockets, delivering multi-language transcripts as JSON.
• Embedding Retrieval - Retrieves semantically similar past discussions by converting conversation text into high-dimensional vector embeddings.
• Tool Invocation Handlers - Handles the invocation and execution of registered custom functions called by an AI assistant during conversations.
• Memory Structuring - Uses large language models to analyze raw captured activity and structure it into meaningful, persistent memory entries.
• Personal Data Retrieval - Utilizes vector search and metadata filters to retrieve relevant past conversations and people from a user's private history.
• Real-Time Speech Transcription - Converts live audio streams from wearable hardware into text with speaker identification and chronological ordering.
• Real-Time Speaker Identifiers - Distinguishes between different speakers by assigning labels to transcribed words during live audio streaming.
• Speaker Verification - Matches live voice embeddings against stored profiles to authenticate and distinguish between different speakers.
• Transcript Summarizers - Summarizes audio transcripts using AI models to extract overviews and action items.
• Structured Data Extraction - Uses AI to identify and extract structured factual data about user preferences and habits from conversations.
• Semantic Vector Search - Retrieves relevant past interactions and facts using vector embeddings and semantic similarity search.
• Wearable AI Platforms - Integrates wearable hardware, real-time sensing, and LLMs to provide a comprehensive conversational AI platform.
• Digital Activity Recorders - Continuously captures and indexes screen and audio activity to create a searchable digital archive of interactions.
• Transcript-Vector Hybrid Storage - Implements a specialized storage architecture combining full-text conversation transcripts with semantic vector summaries for dual-mode retrieval.
• Conversation Insights - Extracts titles, summaries, and personal facts from conversation transcripts using generative AI.
• Agent-Specific Query Routing - Directs user queries to specific processing paths or AI agent personas based on the required context.
• Vector Databases - Employs a vector database to query high-dimensional embeddings and metadata filters for memory retrieval.
• Vector Memory Stores - Utilizes a hybrid vector memory store to maintain long-term context and retrieve semantically similar past interactions.
• Vector Storage - Utilizes a vector database to store high-dimensional embeddings of conversations for efficient semantic retrieval.
• Screen and Audio Capture - Records synchronized audio and screen content across devices to create a persistent record of user activity.
• Real-Time Transcription Pipelines - Implements a real-time data pipeline that streams raw audio over BLE for immediate transcription and event emission.
• Wearable Hardware Development Kits - Supplies a comprehensive hardware development kit including schematics and firmware tools for building AI wearables.
• PCM Audio Streaming - Transmits raw PCM audio from a wearable device over BLE using configurable codecs and packet fragmentation.
• Bidirectional Audio Transports - Streams live audio from hardware to a backend using a combination of Bluetooth Low Energy and WebSockets.
• RESTful Memory Administration APIs - Provides REST endpoints for the programmatic management of AI memories, conversations, and action items.
• Voice Activity Detection - Identifies speech segments within audio streams to trigger downstream processing and manage conversational turns.
• Custom Extension Integrations - Offers interfaces for developers to create custom tools and applications that extend AI agent capabilities.
• Query Path Routing - Classifies user questions into distinct paths to determine if personal memory retrieval or specific persona logic is required.
• External API Tool Exposures - Exposes external service functions as callable tools for AI assistants via a manifest file.
• AI Chat Actions - Defines custom functions that can be invoked directly within a chat conversation to perform external actions.
• Streaming Chat Responses - Sends AI-generated response text and tool-execution status to the client incrementally using Server-Sent Events.
• Persona Configurations - Allows configuration of specific AI personalities and behavioral constraints via a web-based interface.
• Spoken Language Detection - Automatically identifies the spoken language from an audio stream without requiring a preset language code.
• Conversation Summaries - Produces concise AI-generated condensations and actionable items from recorded conversation threads.
• Persona Interaction Interfaces - Provides interfaces for users to engage in themed conversations with specific AI character personas.
• Persona Emulation - Generates AI responses using specific character prompts and curated tool sets to maintain behavioral consistency.
• Streaming Transcription Events - Streams real-time transcription arrays and system status updates to the client via WebSockets.
• Silence-Based Conversation Triggers - Uses audio silence detection to determine the end of a conversation and trigger post-processing.
• Speaker Identification Frameworks - Matches voice input against stored speech profiles to identify specific users during transcription.
• Voice-Driven Interfaces - Enables the extraction of action items and management of calendars and issue trackers through natural language voice input.
• Source Citations - Generates AI responses with numbered citations that provide verifiable references to the original data sources.
• Action Item Tracking - Extracts tasks from conversation transcripts and allows manual creation to manage user obligations.
• Voice-to-Issue Workflows - Transcribes spoken feedback into professional GitHub issues with AI-generated titles and descriptions.
• Natural Language Task Creation - Enables the creation and updating of tasks from natural language requests within a conversation.
• User Preference Management - Captures and persists timeless information regarding user preferences and personal relationships for AI context.
• Conversation Processing Pipelines - Routes transcribed text through an event-driven pipeline to generate memories and summaries while buffering during outages.
• Document Storage Managers - Manages the storage and retrieval of full conversation transcripts and associated metadata using a document database.
• Full Text Search - Implements keyword-based full-text indexing and retrieval across captured conversational data.
• Event-Driven Memory Triggers - Implements automated triggers to capture or process memories based on runtime system events.
• Interaction History - Persists a chronological record of user interactions and associated media files in a cloud database.
• Automated Task Generation - Analyzes captured conversation data to automatically produce concise summaries and actionable task lists.
• Conversation Memory SDKs - Provides software development kits and interfaces for building custom applications that process memories and conversations.
• Community Extension Integrators - Provides a framework for installing and running third-party community applications to extend system features.
• Cloud Storage - Persists binary data including audio recordings and photos in cloud-based storage.
• Serverless Function Executions - Utilizes serverless function execution via Modal for scalable, on-demand processing of audio and conversation data.
• Cloud or Self-Hosted Deployments - Provides flexible deployment options allowing users to choose between a managed cloud service or a self-hosted backend.
• BLE Device Management - Implements a standard interface for BLE communication and data streaming to integrate wearable hardware.
• Over-the-Air Device Updates - Supports delivering and installing system software updates to wearable hardware remotely via a companion app.
• Open-Source Hardware Sync - Connects open-source AI wearables to a mobile application for continuous audio and screen data capture.
• Websocket Connection Managers - Maintains persistent WebSocket connections using keepalive pings and inactivity timeouts for reliable real-time communication.
• WebSocket Clients and Servers - Maintains persistent bidirectional WebSocket connections for streaming live transcripts and system events between client and server.
• Cross-Device Data Availability - Ensures seamless access and synchronization of captured data across wearable, mobile, and desktop devices.
• Event Bus Architectures - Uses an asynchronous event bus to route completed conversation segments for automated summarization and memory creation.
• Citation-to-Source Navigators - Provides UI components that allow users to navigate from numbered citations directly to the source conversation context.
• Integration SDKs - Offers SDKs and REST endpoints for building third-party applications using captured conversation data.

Star history