Awesome GitHub RepositoriesReal-Time Speech Processing

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

Vosk is an offline speech-to-text engine and API that converts spoken audio into text locally on a device. It provides a cross-platform speech toolkit with language bindings for integrating voice recognition into server environments, Android, iOS, and Raspberry Pi. The project includes a speaker identification tool to distinguish between different voices and an acoustic model trainer for building custom neural network models. These training tools enable speech feature extraction and model accuracy evaluation to improve recognition for specialized domains. The system supports real-time audio

Omi is an open-source wearable AI platform that captures audio and screen data to provide real-time conversational assistance and memory. It integrates a wearable hardware development kit with a vector memory database and large language model capabilities to create a persistent digital record of user interactions. The platform is distinguished by its BLE audio streaming pipeline, which transmits raw audio from wearable hardware for real-time transcription and speaker identification. It utilizes a plugin-based agent tool framework that allows AI assistants to autonomously invoke custom functio

This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages. The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware with

Ten Framework is a multimodal large language model agent framework designed for building low-latency conversational agents. It integrates voice, text, and visual inputs in real time to facilitate human interaction. The project includes a real-time speech processing pipeline for streaming transcription, voice activity detection, and speaker diarization. It also features an avatar synchronization engine that coordinates character lip animations and visual outputs with synthesized speech. The framework covers edge AI deployment through containerized packaging and direct integration with embedde

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

RealtimeSTT is a local speech-to-text engine and real-time automatic speech recognition server. It utilizes transformer-based recognition and omnilingual pipelines to convert live audio streams into text, providing a WebSocket-based streaming API for raw PCM audio transmission. The project is distinguished by a dual-backend transcription pipeline that uses a lightweight engine for immediate partial suggestions and a heavier model for final high-accuracy results. It includes a wake word detection system to trigger recording and employs a shared-resource inference model to distribute heavy spee

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

Moonshine is a complete on-device voice interface toolkit that provides speech recognition, text-to-speech synthesis, phonetic processing, speaker diarization, and intent recognition, all running locally on edge hardware without any cloud dependency. It executes quantized neural networks for speech and language tasks directly on the device, enabling fully offline conversational AI capabilities. The toolkit distinguishes itself by orchestrating multi-turn spoken exchanges through a conversational flow manager that maintains context across interactions and manages branching dialog flows. It inc

This project is a Chinese automatic speech recognition framework and deep learning system designed to convert spoken Chinese audio into written text. It functions as a toolkit for training, evaluating, and deploying speech-to-text models, utilizing a specialized pinyin-to-text converter that transforms phonetic sequences into Chinese characters using a probability graph model. The system is distinguished by its deployment flexibility, offering a dockerized recognition server that provides transcription capabilities as a remote API. It supports high-performance streaming through a gRPC speech-

Silero VAD is a voice activity detection model and deep learning speech classifier designed to distinguish human speech from silence across diverse languages and noisy environments. It functions as a pre-trained neural network capable of identifying speech segments within both static audio recordings and real-time data streams. The project includes a language identification tool for classifying spoken languages and a framework for fine-tuning audio models. It provides utilities for optimizing detection thresholds using validation datasets and retraining the model with custom labeled audio to

MeloTTS is an open-source text-to-speech library that generates natural-sounding speech across six languages, with the ability to mix two languages within a single utterance. Its architecture combines a token-based text frontend with a language-agnostic acoustic model, enabling it to handle bilingual code-switching and produce streaming audio output in real time. The system is designed to run efficiently on standard CPU hardware without requiring a dedicated GPU, using a lightweight neural network for real-time inference. It supports English, Spanish, French, Chinese, Japanese, and Korean, an

Este proyecto es un framework para construir asistentes de voz locales y un servidor de streaming de audio en tiempo real. Funciona como un motor de inferencia contenedorizado y un pipeline de voz multilingüe que orquesta componentes de conversión de voz a texto, modelos de lenguaje y conversión de texto a voz para transformar la entrada hablada en salida hablada. El sistema se distingue por su uso de streaming bidireccional basado en WebSocket para interacciones de baja latencia. Cuenta con un sistema de detección de actividad de voz que gestiona los límites del habla y maneja interrupciones del usuario durante la reproducción del asistente. También admite la clonación de voz personalizada mediante preajustes de audio y la capacidad de intercambiar puntos de control de modelos o APIs externas para el reconocimiento y la síntesis. El framework cubre una amplia superficie de capacidades, incluyendo buffering de audio asíncrono, gestión de turnos basada en eventos y ejecución de herramientas basada en esquemas. Proporciona soporte para la gestión de conversaciones multilingües y ejecuta sesiones concurrentes mediante aislamiento de pipelines basado en hilos. El proyecto está disponible como imágenes de contenedor optimizadas para arquitecturas x86 y ARM64.

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

Cheetah is an LLM technical interview assistant composed of a native macOS application and a browser extension. It provides real-time coding and answering suggestions during technical interviews by combining live audio transcription with web-based context extraction. The system functions as a real-time interview coach that converts spoken questions into text using on-device speech-to-text processing. It uses a browser-integrated DOM scraper to extract live code and console logs, allowing the AI to analyze the current coding state and generate technical solutions based on the specific environm

PocketSphinx is an offline speech recognition engine that converts raw audio from files or live microphone streams into written text without requiring a network connection. It functions as a speech-to-text library, a real-time transcription engine, and a voice command processor, capable of detecting and transcribing spoken commands from continuous audio streams with configurable acoustic and language models. The engine uses weighted finite-state transducers to represent acoustic, phonetic, and language models as a single search graph for efficient decoding. It employs fixed-point acoustic mod

Qwen2.5-Omni is an omnichannel multimodal large language model designed to process and generate content across text, audio, vision, and video. It functions as a real-time speech AI, utilizing an end-to-end architecture to maintain synchronous voice conversations with low-latency responses. The project emphasizes efficiency through quantized edge models, allowing for local inference on mobile hardware and resource-constrained devices. It employs 4-bit weight quantization, CPU-based process offloading, and on-demand weight loading to reduce GPU memory requirements. The system integrates specia

RealtimeTTS es un motor de texto a voz (TTS) en tiempo real y procesador de flujos diseñado para convertir texto o flujos de tokens en reproducción de audio con una latencia mínima. Proporciona una interfaz programática para gestionar flujos de audio, el progreso de la síntesis y la integración de motores de voz locales o basados en la nube. El sistema incluye una herramienta de clonación de voz neuronal que genera voz sintética extrayendo características acústicas de muestras de audio de referencia. Utiliza una abstracción basada en proveedores para enrutar las solicitudes de síntesis a través de diferentes modelos neuronales y APIs en la nube. El proyecto cubre una serie de dominios funcionales que incluyen control de reproducción asíncrona, síntesis de audio a partir de flujos de tokens y flujos de trabajo de sincronización de voz que activan callbacks a nivel de palabra o carácter. También admite la exportación de audio sintetizado directamente a archivos WAV para almacenamiento persistente.

RealtimeVoiceChat is a conversational AI gateway and voice chat interface that enables real-time spoken interactions with large language models. It functions as a streaming audio pipeline that integrates speech-to-text transcription and text-to-speech synthesis to facilitate natural voice conversations. The system utilizes WebSocket-based audio streaming and an asynchronous text-to-speech pipeline to maintain low latency. It includes specialized conversational controls such as silence-based turn detection and client-side interrupt handling, which stops active audio playback when new user spee