Awesome GitHub RepositoriesMultilingual Speech Translation

This project is a speech recognition and translation engine that utilizes a sequence-to-sequence transformer architecture to convert audio into text. It is built upon a weakly supervised learning framework, which leverages large-scale, unlabelled audio-transcript data to create generalized speech representations capable of performing simultaneous transcription, language identification, and translation. The system distinguishes itself through a unified multi-task modeling approach that shares token sequences across different objectives, allowing it to handle diverse languages and vocabularies

LiveKit is a comprehensive framework for building and orchestrating real-time, multimodal AI agents that interact with users through voice, video, and text. It provides a centralized, event-driven architecture to manage the entire lifecycle of automated participants, from initialization and session state management to graceful shutdown. By utilizing a selective forwarding unit, the platform efficiently routes media streams between participants and agents, ensuring low-latency communication and secure, token-based authentication for all connections. The platform distinguishes itself through it

Buzz is a desktop application that provides a local speech-to-text engine for transcribing and translating audio and video files. By leveraging local machine inference, the software ensures data privacy and offline performance, removing the need for cloud connectivity during media processing. The application distinguishes itself through a modular plugin architecture that allows for the integration of custom functionality, such as content summarization and automated text formatting, without modifying the core codebase. It also features a speaker diarization pipeline that identifies and labels

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

Handy is a local speech-to-text automation tool designed to convert spoken audio into text and inject it directly into active desktop applications. By running machine learning models entirely on the host hardware, it provides a private, offline-first environment for dictation and command execution. The system functions as a background service that manages microphone input, transcription state, and text output, enabling hands-free typing across various software environments. The project distinguishes itself through a modular pipeline that integrates local language models for post-transcription

Pipecat is a framework and software development kit for building real-time multimodal AI agents and speech-to-speech systems. It utilizes a frame-based data pipeline to route audio, video, and text through a modular sequence of processors, enabling the orchestration of low-latency conversational AI. The project is distinguished by its ability to coordinate complex multimodal services, including speech-to-text, language models, and text-to-speech, within a single pipeline. It features semantic voice activity detection for natural turn-taking, state-machine conversation flows for dialogue manag

PaddleSpeech is a comprehensive toolkit of neural models for speech recognition, synthesis, and translation built on the PaddlePaddle deep learning framework. It provides a collection of frameworks and tools for converting spoken audio into written text, synthesizing natural audio from text, and performing direct speech translation. The toolkit includes specialized capabilities for keyword spotting to detect trigger words and speaker verification systems that extract unique voiceprints to identify and distinguish between individuals. It also features end-to-end translation tools that map audi

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

whisper-jax es una implementación de alto rendimiento del modelo de reconocimiento automático de voz Whisper, reescrita utilizando el framework JAX. Está diseñada para una inferencia acelerada y utiliza la compilación XLA para optimizar la ejecución del modelo en aceleradores de hardware. El proyecto se centra en la transcripción optimizada para TPU para lograr un alto rendimiento y velocidad. Incluye un pipeline de traducción de pesos que convierte los parámetros del modelo preentrenado de PyTorch en arrays compatibles con JAX. El sistema admite la transcripción de audio a texto, la traducción de voz en varios idiomas y la generación de marcas de tiempo de audio. Permite el procesamiento de audio por lotes y escala el rendimiento mediante el procesamiento por lotes paralelo a los datos y la partición de tensores paralela al modelo. El proyecto proporciona un método para desplegar el modelo de transcripción como un endpoint de inferencia remoto con una interfaz web.

Qwen3-Omni is an omni-modal large language model designed to process and generate text, audio, images, and video within a single unified neural architecture. It functions as a real-time voice assistant and multimodal AI agent capable of reasoning across different media types and executing external tool-calling functions via APIs. The system supports low-latency conversational AI through autoregressive token streaming and natural turn-taking. It enables multilingual speech translation and generation across dozens of languages, featuring customizable speaker profiles and tones. The model's cap

Este proyecto es un toolkit integral para el reconocimiento de voz, síntesis y procesamiento de audio en el dispositivo, diseñado específicamente para Apple Silicon. Proporciona un framework para construir agentes de voz full-duplex en tiempo real que operan completamente offline, aprovechando la aceleración de hardware nativa para mantener el rendimiento y la privacidad. Al utilizar modelos de machine learning optimizados, la biblioteca permite la ejecución local de tareas de audio complejas sin depender de servicios externos en la nube. La biblioteca se distingue por su enfoque especializado en la interacción de voz local de alto rendimiento. Incluye una orquestación sofisticada para pipelines de audio en streaming, permitiendo la transcripción en tiempo real, síntesis de voz y clonación de voz con baja latencia. El sistema está diseñado para manejar conversaciones interactivas continuas, presentando mecanismos integrados para evitar bucles de retroalimentación de audio y gestionar sesiones de streaming persistentes. Más allá de la interacción central, el proyecto ofrece un amplio conjunto de capacidades de mejora y gestión de audio. Admite procesamiento de señales avanzado, incluyendo separación de fuentes, reducción de ruido y sobremuestreo de audio, junto con herramientas para diarización de hablantes y extracción de embeddings. El framework también proporciona amplias utilidades de gestión de modelos, como controles de cuantización, gestión de memoria y soporte para la carga de pesos de modelos personalizados, asegurando que los desarrolladores puedan equilibrar la velocidad de procesamiento y el consumo de recursos en hardware local. El proyecto incluye una interfaz de línea de comandos para ejecutar tareas de audio y convertir pesos de modelos en formatos optimizados. También expone endpoints HTTP y WebSocket para facilitar la integración con interfaces estándar de la industria.