huggingface/speech-to-speech
Speech To Speech
该项目是一个用于构建本地语音助手和实时音频流服务器的框架。它作为一个容器化推理引擎和多语言语音流水线，编排语音转文字 (STT)、语言模型和文字转语音 (TTS) 组件，将语音输入转换为语音输出。
该系统以其使用基于 WebSocket 的双向流来实现低延迟交互而著称。它具有一个语音活动检测系统，可管理语音边界并处理助手播放期间的用户打断。它还支持通过音频预设进行自定义语音克隆，以及交换模型检查点或外部 API 进行识别和合成的能力。
该框架涵盖了广泛的功能面，包括异步音频缓冲、事件驱动的轮次管理和基于 Schema 的工具执行。它支持多语言对话管理，并通过基于线程的流水线隔离运行并发会话。
该项目提供针对 x86 和 ARM64 架构优化的容器镜像。
Features
- Voice Agents - Provides a framework for building local voice agents that convert spoken input into spoken output.
- Voice Assistants - Provides a framework for building local voice assistants using open-source models for speech-to-speech interaction.
- Turn Event Emitters - Emits events when user speech boundaries are detected to trigger the corresponding transcription and synthesis stages.
- Voice Activity Detection - Uses silence and duration thresholds to automatically identify speech segments within audio streams.
- Voice Pipelines - Coordinates the sequence of voice detection, transcription, language processing, and synthesis into a fluid loop.
- Voice Interaction Management - Coordinates the entire voice interaction loop, from speech detection and recognition to response synthesis.
- Model Provider Integrations - Connects to diverse language model providers through a unified interface supporting local and cloud-based APIs.
- Real-Time Transcription - Provides instantaneous conversion of live audio streams into text transcripts displayed in real-time.
- Speech Boundary Detection - Identifies exact start and end timestamps of human speech to trigger transcription and manage conversational turns.
- Modular Pipeline Orchestrators - Provides a modular orchestrator that separates voice activity detection, transcription, and synthesis into independent processing components.
- Real-Time Speech Processing - Coordinates low-latency workflows sequencing voice activity detection, transcription, language processing, and synthesis.
- Speech-to-Text and Text-to-Speech Integrations - Provides a configurable pipeline for integrating speech-to-text, language models, and text-to-speech backends.
- Voice Activity Detection - Identifies speech boundaries to manage conversational turn-taking and handle user barge-in interruptions.
- Bidirectional WebSocket Streaming - Implements bidirectional WebSocket streaming for low-latency exchange of raw audio and text data between client and server.
- Audio Streaming Servers - Implements a dedicated audio streaming server for bidirectional raw audio delivery with low-latency turn-taking.
- Bidirectional Audio Transports - Implements WebSocket-based bidirectional audio transports for low-latency turn-taking and live transcription.
- Voice Interaction Interfaces - Provides a conversational interface that manages turn-taking and user interruptions in real-time audio streams.
- Containerized Deployments - Packages the voice agent environment and dependencies into portable container images for consistent deployment.
- Threshold Tunings - Allows adjustment of speech duration and silence thresholds to balance responsiveness with accurate turn segmentation.
- Model Checkpoint Swapping - Enables integrating different model checkpoints or external APIs to optimize for specific hardware or latency requirements.
- Voice Cloning Tools - Generates synthetic speech that mimics specific speakers using custom audio files or voice presets.
- Local Voice Control Interfaces - Executes full speech-to-speech pipelines on local hardware using accelerators to minimize interaction latency.
- Local Inference Engines - Acts as a local inference engine for running speech-to-speech models on consumer hardware.
- Speech-to-Speech Translation - Implements a multilingual pipeline that detects spoken languages and converts audio to audio across different tongues.
- Tool Call Executions - Extracts structured tool calls from model output using JSON schemas to trigger external actions.
- Multilingual Audio Processing - Features a multilingual audio processing chain that handles language detection and audio-to-audio conversion.
- Multilingual Conversational Interaction - Supports automatic spoken language detection and manages transcription and synthesis across multiple languages.
- Barge-In Handlers - Implements handlers that stop current speech output immediately when a user barge-in event is detected.
- Tool-Calling Schemas - Implements schema-based tool calling to allow language models to trigger external functions and retrieve data during voice conversations.
- Real-Time Voice Protocols - Exposes a WebSocket endpoint supporting live transcription and turn-taking based on standardized real-time specifications.
- Audio Buffers - Uses circular memory buffers for low-latency capture and real-time processing of raw audio input.
- Model Agnostic Interfaces - Decouples high-level API calls from specific model implementations, allowing local transformers or cloud APIs to be used interchangeably.
