KoljaB/RealtimeSTT
RealtimeSTT
RealtimeSTT is a local speech-to-text engine and real-time automatic speech recognition server. It utilizes transformer-based recognition and omnilingual pipelines to convert live audio streams into text, providing a WebSocket-based streaming API for raw PCM audio transmission.
The project is distinguished by a dual-backend transcription pipeline that uses a lightweight engine for immediate partial suggestions and a heavier model for final high-accuracy results. It includes a wake word detection system to trigger recording and employs a shared-resource inference model to distribute heavy speech models across multiple concurrent user sessions.
Its broader capabilities cover audio processing tasks such as voice activity detection, speaker diarization, and speaker emotion detection. The system also supports real-time speech translation, automated system input routing to simulate keyboard typing, and an extensible engine factory for adding new transcription backends.
The server includes dedicated health and performance monitoring endpoints to track active sessions, inference latency, and worker utilization.
Features
- Real-Time Speech Processing - Provides a complete real-time pipeline for converting live audio streams into text using local models.
- Audio Transcription - Converts raw audio chunks from files or websockets into text by resampling audio to the required processing rate.
- Real-Time Transcription - Performs instantaneous conversion of live microphone audio streams into text transcripts via a persistent connection.
- Transcription APIs - Provides a WebSocket-based streaming server that offers programmatic transcription capabilities for integration into external applications.
- Transformer-Based ASR - Converts live audio to text using pre-trained transformer model families with specific generation settings.
- Local Speech-to-Text - Runs transcription inference on local hardware to ensure data privacy and remove external API dependencies.
- Automatic Speech Recognition - Utilizes transformer-based recognition and omnilingual pipelines for high-accuracy real-time transcription.
- Dual-Model Transcription Pipelines - Produces fast, preliminary transcriptions using a lightweight model while a larger model processes the final result.
- Incremental Transcription Previews - Generates incremental text results during active speech to show immediate progress before final transcription completes.
- Streaming Transcription Inference - Processes audio frames through a low-latency inference pipeline for immediate transcription previews.
- Hybrid Precision Pipelines - Utilizes a dual-backend pipeline with a lightweight engine for immediate partials and a heavy model for final accuracy.
- Speech Processing WebSocket Servers - Implements a WebSocket server that exposes real-time speech-to-text capabilities.
- Wake Word Detection - Monitors audio streams for specific activation phrases to trigger recording only after a wake word is spoken.
- Realtime Speech Translation - Provides real-time conversion of spoken audio from one language into text in another language.
- Real-Time Speech Translation - Converts spoken audio from one language into text in another language in real time.
- Speaker Diarization - Distinguishes between multiple voices in a single audio stream to attribute transcribed text to specific speakers.
- Speech-to-Text Integrations - Utilizes an omnilingual pipeline that supports various model sizes and compute types for multi-language transcription.
- Voice Activity Detection - Detects speech boundaries to automatically trigger recording and transcription cycles using sensitivity filters.
- Runtime Resource Sharing - Loads heavy speech models into memory once and shares them across multiple concurrent user sessions to minimize overhead.
- Multi-User Session Isolation - Ships a multi-user web server that isolates sessions while sharing inference resources for remote transcription.
- PCM Audio Streaming - Accepts raw PCM audio over WebSockets and returns incremental transcription results.
- Audio Transcription WebSockets - Implements a WebSocket connection to transmit live PCM audio for real-time transcription results.
- Inference Stream Multiplexing - Manages multiple concurrent user sessions by isolating audio buffers while sharing model weight memory for inference.
- Inference Session Isolation - Isolates audio buffers and transcription states for multiple simultaneous users while sharing a single inference engine.
- Transcription Engine Adapters - Implements modular interfaces that allow swapping different speech-to-text engines without modifying core application logic.
- Split-Pipeline Transcription - Allows using separate transcription engines for partial and final results to balance processing speed with accuracy.
