Blaizzy/mlx-audio
Mlx Audio
mlx-audio is an audio processing toolkit built on Apple MLX that provides speech transcription, text-to-speech synthesis, voice cloning, and audio source separation using local models. It offers an OpenAI-compatible REST API and web interface for running audio generation and transcription tasks, enabling drop-in integration with existing tools that follow that endpoint structure.
The toolkit supports text-prompted audio source separation, allowing specific sounds to be isolated from mixed recordings based on natural language descriptions. It also provides voice cloning from a short reference audio sample, speech enhancement through noise reduction, and voice activity detection with speaker diarization to distinguish between different speakers in recordings.
Additional capabilities include speech-to-text transcription with word-level timestamp alignment, streaming audio generation that outputs results incrementally, and model weight quantization to reduce memory footprint and accelerate inference. The system manages multiple models through a unified interface and supports WebSocket audio transport for low-latency communication.
Features
- Speech Processing Toolkits - An audio toolkit built on Apple MLX for speech transcription, text-to-speech, voice cloning, and source separation.
- OpenAI-Compatible APIs - Exposes audio processing capabilities through an OpenAI-compatible REST API for drop-in integration.
- Audio Source Separation Models - Isolates specific sounds from mixed audio files using natural language text prompts.
- Text-Prompted Separators - Isolates specific sounds from mixed audio files using natural language text prompts.
- Audio Model Hubs - Provides a unified interface for loading and switching between multiple audio processing models.
- Speech to Text Transcription - Transcribes spoken audio into written text using multilingual speech recognition with word-level timestamps.
- Text-to-Speech - Provides multilingual text-to-speech synthesis with voice cloning and streaming capabilities.
- Voice Cloning - Replicates a speaker's voice from a short reference audio sample for personalized speech generation.
- Streaming Audio Generators - Ships a streaming audio generation pipeline that outputs results incrementally for low-latency playback.
- Word-Level Timestamps - Provides word-level timestamp alignment for transcribed speech, mapping each word to precise audio positions.
- Speaker Diarizers - Detects speech segments and distinguishes between different speakers in audio recordings.
- Model Quantization - Reduces model weight precision to decrease size and accelerate inference for audio processing models.
- Weight Quantization - Loads model weights in reduced-precision formats at runtime to decrease memory footprint and accelerate inference.
- Speech Enhancers - Cleans noisy audio recordings and distinguishes between different speakers for clearer speech output.
- Voice Activity Detectors - Detects speech segments and distinguishes between different speakers in audio recordings.
- Audio Streaming Pipelines - Processes audio in chunks through a chain of models for real-time generation and transcription.
- Audio API Servers - Provides an interactive web UI and OpenAI-compatible REST API for audio generation and transcription.
