huggingface/speech-to-speech

Features

• Voice Agents - Provides a framework for building local voice agents that convert spoken input into spoken output.
• Voice Assistants - Provides a framework for building local voice assistants using open-source models for speech-to-speech interaction.
• Turn Event Emitters - Emits events when user speech boundaries are detected to trigger the corresponding transcription and synthesis stages.
• Voice Activity Detection - Uses silence and duration thresholds to automatically identify speech segments within audio streams.
• Voice Pipelines - Coordinates the sequence of voice detection, transcription, language processing, and synthesis into a fluid loop.
• Voice Interaction Management - Coordinates the entire voice interaction loop, from speech detection and recognition to response synthesis.
• Model Provider Integrations - Connects to diverse language model providers through a unified interface supporting local and cloud-based APIs.
• Real-Time Transcription - Provides instantaneous conversion of live audio streams into text transcripts displayed in real-time.
• Speech Boundary Detection - Identifies exact start and end timestamps of human speech to trigger transcription and manage conversational turns.
• Modular Pipeline Orchestrators - Provides a modular orchestrator that separates voice activity detection, transcription, and synthesis into independent processing components.
• Real-Time Speech Processing - Coordinates low-latency workflows sequencing voice activity detection, transcription, language processing, and synthesis.
• Speech-to-Text and Text-to-Speech Integrations - Provides a configurable pipeline for integrating speech-to-text, language models, and text-to-speech backends.
• Voice Activity Detection - Identifies speech boundaries to manage conversational turn-taking and handle user barge-in interruptions.
• Bidirectional WebSocket Streaming - Implements bidirectional WebSocket streaming for low-latency exchange of raw audio and text data between client and server.
• Audio Streaming Servers - Implements a dedicated audio streaming server for bidirectional raw audio delivery with low-latency turn-taking.
• Bidirectional Audio Transports - Implements WebSocket-based bidirectional audio transports for low-latency turn-taking and live transcription.
• Voice Interaction Interfaces - Provides a conversational interface that manages turn-taking and user interruptions in real-time audio streams.
• Containerized Deployments - Packages the voice agent environment and dependencies into portable container images for consistent deployment.
• Threshold Tunings - Allows adjustment of speech duration and silence thresholds to balance responsiveness with accurate turn segmentation.
• Model Checkpoint Swapping - Enables integrating different model checkpoints or external APIs to optimize for specific hardware or latency requirements.
• Voice Cloning Tools - Generates synthetic speech that mimics specific speakers using custom audio files or voice presets.
• Local Voice Control Interfaces - Executes full speech-to-speech pipelines on local hardware using accelerators to minimize interaction latency.
• Local Inference Engines - Acts as a local inference engine for running speech-to-speech models on consumer hardware.
• Speech-to-Speech Translation - Implements a multilingual pipeline that detects spoken languages and converts audio to audio across different tongues.
• Tool Call Executions - Extracts structured tool calls from model output using JSON schemas to trigger external actions.
• Multilingual Audio Processing - Features a multilingual audio processing chain that handles language detection and audio-to-audio conversion.
• Multilingual Conversational Interaction - Supports automatic spoken language detection and manages transcription and synthesis across multiple languages.
• Barge-In Handlers - Implements handlers that stop current speech output immediately when a user barge-in event is detected.
• Tool-Calling Schemas - Implements schema-based tool calling to allow language models to trigger external functions and retrieve data during voice conversations.
• Real-Time Voice Protocols - Exposes a WebSocket endpoint supporting live transcription and turn-taking based on standardized real-time specifications.
• Audio Buffers - Uses circular memory buffers for low-latency capture and real-time processing of raw audio input.
• Model Agnostic Interfaces - Decouples high-level API calls from specific model implementations, allowing local transformers or cloud APIs to be used interchangeably.

Star history