Awesome GitHub RepositoriesReal-Time Speech Processing

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

Vosk is an offline speech-to-text engine and API that converts spoken audio into text locally on a device. It provides a cross-platform speech toolkit with language bindings for integrating voice recognition into server environments, Android, iOS, and Raspberry Pi. The project includes a speaker identification tool to distinguish between different voices and an acoustic model trainer for building custom neural network models. These training tools enable speech feature extraction and model accuracy evaluation to improve recognition for specialized domains. The system supports real-time audio

Omi is an open-source wearable AI platform that captures audio and screen data to provide real-time conversational assistance and memory. It integrates a wearable hardware development kit with a vector memory database and large language model capabilities to create a persistent digital record of user interactions. The platform is distinguished by its BLE audio streaming pipeline, which transmits raw audio from wearable hardware for real-time transcription and speaker identification. It utilizes a plugin-based agent tool framework that allows AI assistants to autonomously invoke custom functio

This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages. The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware with

Ten Framework is a multimodal large language model agent framework designed for building low-latency conversational agents. It integrates voice, text, and visual inputs in real time to facilitate human interaction. The project includes a real-time speech processing pipeline for streaming transcription, voice activity detection, and speaker diarization. It also features an avatar synchronization engine that coordinates character lip animations and visual outputs with synthesized speech. The framework covers edge AI deployment through containerized packaging and direct integration with embedde

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

RealtimeSTT is a local speech-to-text engine and real-time automatic speech recognition server. It utilizes transformer-based recognition and omnilingual pipelines to convert live audio streams into text, providing a WebSocket-based streaming API for raw PCM audio transmission. The project is distinguished by a dual-backend transcription pipeline that uses a lightweight engine for immediate partial suggestions and a heavier model for final high-accuracy results. It includes a wake word detection system to trigger recording and employs a shared-resource inference model to distribute heavy spee

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

Moonshine is a complete on-device voice interface toolkit that provides speech recognition, text-to-speech synthesis, phonetic processing, speaker diarization, and intent recognition, all running locally on edge hardware without any cloud dependency. It executes quantized neural networks for speech and language tasks directly on the device, enabling fully offline conversational AI capabilities. The toolkit distinguishes itself by orchestrating multi-turn spoken exchanges through a conversational flow manager that maintains context across interactions and manages branching dialog flows. It inc

This project is a Chinese automatic speech recognition framework and deep learning system designed to convert spoken Chinese audio into written text. It functions as a toolkit for training, evaluating, and deploying speech-to-text models, utilizing a specialized pinyin-to-text converter that transforms phonetic sequences into Chinese characters using a probability graph model. The system is distinguished by its deployment flexibility, offering a dockerized recognition server that provides transcription capabilities as a remote API. It supports high-performance streaming through a gRPC speech-

Silero VAD is a voice activity detection model and deep learning speech classifier designed to distinguish human speech from silence across diverse languages and noisy environments. It functions as a pre-trained neural network capable of identifying speech segments within both static audio recordings and real-time data streams. The project includes a language identification tool for classifying spoken languages and a framework for fine-tuning audio models. It provides utilities for optimizing detection thresholds using validation datasets and retraining the model with custom labeled audio to

MeloTTS is an open-source text-to-speech library that generates natural-sounding speech across six languages, with the ability to mix two languages within a single utterance. Its architecture combines a token-based text frontend with a language-agnostic acoustic model, enabling it to handle bilingual code-switching and produce streaming audio output in real time. The system is designed to run efficiently on standard CPU hardware without requiring a dedicated GPU, using a lightweight neural network for real-time inference. It supports English, Spanish, French, Chinese, Japanese, and Korean, an

Ce projet est un framework pour construire des assistants vocaux locaux et un serveur de streaming audio en temps réel. Il fonctionne comme un moteur d'inférence conteneurisé et un pipeline vocal multilingue qui orchestre des composants de reconnaissance vocale (STT), des modèles de langage et de synthèse vocale (TTS) pour convertir une entrée parlée en sortie parlée. Le système se distingue par son utilisation du streaming bidirectionnel basé sur WebSocket pour des interactions à faible latence. Il dispose d'un système de détection d'activité vocale qui gère les limites de la parole et traite les interruptions de l'utilisateur pendant la lecture de l'assistant. Il prend également en charge le clonage vocal personnalisé via des préréglages audio et la possibilité d'échanger des checkpoints de modèles ou des API externes pour la reconnaissance et la synthèse. Le framework couvre une large surface de capacités, incluant la mise en mémoire tampon audio asynchrone, la gestion des tours de parole pilotée par événements et l'exécution d'outils basée sur des schémas. Il fournit un support pour la gestion de conversations multilingues et exécute des sessions simultanées via une isolation de pipeline basée sur les threads. Le projet est disponible sous forme d'images de conteneurs optimisées pour les architectures x86 et ARM64.

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

Cheetah est un assistant d'entretien technique LLM composé d'une application native macOS et d'une extension de navigateur. Il fournit des suggestions de codage et de réponses en temps réel pendant les entretiens techniques en combinant la transcription audio en direct avec l'extraction de contexte basée sur le web. Le système fonctionne comme un coach d'entretien en temps réel qui convertit les questions parlées en texte en utilisant un traitement de reconnaissance vocale sur l'appareil. Il utilise un scraper DOM intégré au navigateur pour extraire le code en direct et les logs de console, permettant à l'IA d'analyser l'état actuel du codage et de générer des solutions techniques basées sur l'environnement spécifique de l'entretien. L'outil prend en charge l'affinement itératif des réponses, permettant aux utilisateurs de mettre à jour les réponses suggérées à mesure que les recruteurs fournissent de nouvelles contraintes ou clarifications. Ces capacités couvrent les domaines plus larges de la préparation aux entretiens techniques et de l'assistance au codage par IA en direct.

PocketSphinx is an offline speech recognition engine that converts raw audio from files or live microphone streams into written text without requiring a network connection. It functions as a speech-to-text library, a real-time transcription engine, and a voice command processor, capable of detecting and transcribing spoken commands from continuous audio streams with configurable acoustic and language models. The engine uses weighted finite-state transducers to represent acoustic, phonetic, and language models as a single search graph for efficient decoding. It employs fixed-point acoustic mod

Qwen2.5-Omni est un modèle de langage multimodal omnicanal conçu pour traiter et générer du contenu textuel, audio, visuel et vidéo. Il fonctionne comme une IA vocale en temps réel, utilisant une architecture de bout en bout pour maintenir des conversations vocales synchrones avec des réponses à faible latence. Le projet met l'accent sur l'efficacité grâce à des modèles de périphérie quantifiés, permettant une inférence locale sur du matériel mobile et des appareils aux ressources limitées. Il emploie une quantification de poids 4 bits, un déchargement des processus sur CPU et un chargement des poids à la demande pour réduire les besoins en mémoire GPU. Le système intègre des encodeurs spécialisés pour analyser les flux de données multimodaux et dispose d'un décodeur en streaming pour la génération vocale en temps réel. Il inclut également des capacités de personnalisation de la voix pour modifier les caractéristiques tonales et le genre de la sortie audio.

RealtimeTTS est un moteur de synthèse vocale (TTS) en temps réel et un processeur de flux conçu pour convertir des flux de texte ou de jetons en lecture audio avec une latence minimale. Il fournit une interface programmatique pour gérer les flux audio, la progression de la synthèse et l'intégration de moteurs de synthèse locaux ou basés sur le cloud. Le système inclut un outil de clonage de voix neuronale qui génère une parole synthétique en extrayant des caractéristiques acoustiques à partir d'échantillons audio de référence. Il utilise une abstraction basée sur des fournisseurs pour router les requêtes de synthèse vers différents modèles neuronaux et API cloud. Le projet couvre une gamme de domaines fonctionnels incluant le contrôle de lecture asynchrone, la synthèse audio par flux de jetons et des workflows de synchronisation vocale qui déclenchent des callbacks au niveau du mot ou du caractère. Il prend également en charge l'exportation de l'audio synthétisé directement vers des fichiers WAV pour un stockage persistant.

RealtimeVoiceChat is a conversational AI gateway and voice chat interface that enables real-time spoken interactions with large language models. It functions as a streaming audio pipeline that integrates speech-to-text transcription and text-to-speech synthesis to facilitate natural voice conversations. The system utilizes WebSocket-based audio streaming and an asynchronous text-to-speech pipeline to maintain low latency. It includes specialized conversational controls such as silence-based turn detection and client-side interrupt handling, which stops active audio playback when new user spee