collabora/WhisperLive
WhisperLive
WhisperLive is a real-time speech-to-text server that converts live audio streams into text using Whisper models. It functions as a backend service that receives microphone input via WebSockets and provides incremental transcriptions with word-level timestamps.
The system utilizes a GPU-accelerated inference engine and a keyword-boosted transcription API to improve the recognition accuracy of domain-specific jargon, acronyms, and product names. It also includes a speaker diarization tool that clusters audio embeddings to identify and label different participants within a recording.
Additional capabilities include high-throughput audio processing via batch inference and TensorRT acceleration, as well as audio signal normalization and recording state control. The service supports live audio captioning through segment-based incremental rendering.
Features
- Real-Time Transcription - Converts live audio streams into text in real time using Whisper models for immediate accessibility.
- Audio Transcription - Provides a backend service that streams microphone input and delivers incremental text transcriptions.
- GPU-Accelerated Inference - Employs a GPU-accelerated inference engine to optimize throughput for multilingual speech recognition.
- Real-Time Speech-to-Text Servers - Functions as a real-time audio transcription server using Whisper models and WebSocket streaming.
- Speaker Diarization - Clusters audio feature vectors to distinguish and segment different speakers within a single audio stream.
- Whisper-Based Engines - Utilizes a Faster-Whisper engine with CTranslate2 backend to optimize transcription speed and memory usage.
- Audio Transcription WebSockets - Uses WebSockets to stream raw PCM audio from the client to the server for real-time processing.
- High-Throughput Transcription - Processes multiple simultaneous audio streams via GPU batching to achieve high transcription throughput.
- Word-Level Timestamps - Produces precise start and end timestamps and confidence scores for every individual word transcribed.
- Inference Acceleration Engines - Implements high-performance inference using TensorRT to accelerate speech-to-text processing speeds.
- Incremental Transcription Previews - Incrementally renders transcribed text segments on the screen as they are emitted by the backend.
- Technical Jargon Optimizations - Improves transcription accuracy for domain-specific technical terms using keyword boosting.
- Inference Batching - Groups multiple concurrent user audio segments into single GPU calls to maximize system throughput.
- Transcription Term Boosts - Provides a mechanism to boost specific technical terms and jargon during the transcription decoding process.
- Live Captioning Integrations - Displays incrementally processed speech as text on screen for real-time live captioning.
- Incremental Text Rendering - Updates the user interface incrementally by appending transcribed text chunks as they are emitted.
