Awesome GitHub RepositoriesMultilingual Speech Translation

This project is a speech recognition and translation engine that utilizes a sequence-to-sequence transformer architecture to convert audio into text. It is built upon a weakly supervised learning framework, which leverages large-scale, unlabelled audio-transcript data to create generalized speech representations capable of performing simultaneous transcription, language identification, and translation. The system distinguishes itself through a unified multi-task modeling approach that shares token sequences across different objectives, allowing it to handle diverse languages and vocabularies

LiveKit is a comprehensive framework for building and orchestrating real-time, multimodal AI agents that interact with users through voice, video, and text. It provides a centralized, event-driven architecture to manage the entire lifecycle of automated participants, from initialization and session state management to graceful shutdown. By utilizing a selective forwarding unit, the platform efficiently routes media streams between participants and agents, ensuring low-latency communication and secure, token-based authentication for all connections. The platform distinguishes itself through it

Buzz is a desktop application that provides a local speech-to-text engine for transcribing and translating audio and video files. By leveraging local machine inference, the software ensures data privacy and offline performance, removing the need for cloud connectivity during media processing. The application distinguishes itself through a modular plugin architecture that allows for the integration of custom functionality, such as content summarization and automated text formatting, without modifying the core codebase. It also features a speaker diarization pipeline that identifies and labels

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

Handy is a local speech-to-text automation tool designed to convert spoken audio into text and inject it directly into active desktop applications. By running machine learning models entirely on the host hardware, it provides a private, offline-first environment for dictation and command execution. The system functions as a background service that manages microphone input, transcription state, and text output, enabling hands-free typing across various software environments. The project distinguishes itself through a modular pipeline that integrates local language models for post-transcription

Pipecat is a framework and software development kit for building real-time multimodal AI agents and speech-to-speech systems. It utilizes a frame-based data pipeline to route audio, video, and text through a modular sequence of processors, enabling the orchestration of low-latency conversational AI. The project is distinguished by its ability to coordinate complex multimodal services, including speech-to-text, language models, and text-to-speech, within a single pipeline. It features semantic voice activity detection for natural turn-taking, state-machine conversation flows for dialogue manag

PaddleSpeech is a comprehensive toolkit of neural models for speech recognition, synthesis, and translation built on the PaddlePaddle deep learning framework. It provides a collection of frameworks and tools for converting spoken audio into written text, synthesizing natural audio from text, and performing direct speech translation. The toolkit includes specialized capabilities for keyword spotting to detect trigger words and speaker verification systems that extract unique voiceprints to identify and distinguish between individuals. It also features end-to-end translation tools that map audi

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

whisper-jax 是使用 JAX 框架重写的 Whisper 自动语音识别模型的高性能实现。它专为加速推理而设计，并使用 XLA 编译来优化硬件加速器上的模型执行。 该项目专注于 TPU 优化的转录，以实现高吞吐量和速度。它包括一个权重转换流水线，将预训练的模型参数从 PyTorch 转换为 JAX 兼容的数组。 该系统支持将音频转录为文本、跨多种语言翻译语音以及生成音频时间戳。它支持批量音频处理，并通过数据并行批处理和模型并行张量分区来扩展性能。 该项目提供了一种将转录模型部署为带有 Web 界面的远程推理端点的方法。

Qwen3-Omni is an omni-modal large language model designed to process and generate text, audio, images, and video within a single unified neural architecture. It functions as a real-time voice assistant and multimodal AI agent capable of reasoning across different media types and executing external tool-calling functions via APIs. The system supports low-latency conversational AI through autoregressive token streaming and natural turn-taking. It enables multilingual speech translation and generation across dozens of languages, featuring customizable speaker profiles and tones. The model's cap

该项目是一个全面的工具包，用于设备端语音识别、合成和音频处理，专为 Apple Silicon 工程设计。它提供了一个框架，用于构建完全离线运行的实时、全双工语音代理，利用原生硬件加速来保持性能和隐私。通过利用优化的机器学习模型，该库实现了复杂音频任务的本地执行，而无需依赖外部云服务。 该库通过其对本地、高性能语音交互的专门关注脱颖而出。它包括用于流式音频流水线的复杂编排，允许以低延迟进行实时转录、语音合成和语音克隆。该系统旨在处理持续的、交互式的对话，具有内置机制来防止音频反馈循环并管理持久的流会话。 除了核心交互外，该项目还提供了一套广泛的音频增强和管理功能。它支持高级信号处理，包括源分离、降噪和音频上采样，以及用于说话人日志记录和嵌入提取的工具。该框架还提供广泛的模型管理工具，例如量化控制、内存管理和对自定义模型权重加载的支持，确保开发者能够在本地硬件上平衡处理速度和资源消耗。 该项目包含一个用于执行音频任务和将模型权重转换为优化格式的命令行接口。它还暴露了 HTTP 和 WebSocket 端点，以促进与标准行业接口的集成。