Blaizzy/mlx-audio
Mlx Audio
mlx-audio is an audio processing toolkit built on Apple MLX that provides speech transcription, text-to-speech synthesis, voice cloning, and audio source separation using local models. It offers an OpenAI-compatible REST API and web interface for running audio generation and transcription tasks, enabling drop-in integration with existing tools that follow that endpoint structure.
The toolkit supports text-prompted audio source separation, allowing specific sounds to be isolated from mixed recordings based on natural language descriptions. It also provides voice cloning from a short reference audio sample, speech enhancement through noise reduction, and voice activity detection with speaker diarization to distinguish between different speakers in recordings.
Additional capabilities include speech-to-text transcription with word-level timestamp alignment, streaming audio generation that outputs results incrementally, and model weight quantization to reduce memory footprint and accelerate inference. The system manages multiple models through a unified interface and supports WebSocket audio transport for low-latency communication.
Features
- Speech Processing Toolkits - An audio toolkit built on Apple MLX for speech transcription, text-to-speech, voice cloning, and source separation.
- OpenAI-Compatible APIs - Exposes audio processing capabilities through an OpenAI-compatible REST API for drop-in integration.
- Audio Source Separation Models - Isolates specific sounds from mixed audio files using natural language text prompts.
- Text-Prompted Separators - Isolates specific sounds from mixed audio files using natural language text prompts.
- Audio Model Hubs - Provides a unified interface for loading and switching between multiple audio processing models.
- Speech to Text Transcription - Transcribes spoken audio into written text using multilingual speech recognition with word-level timestamps.
- Text-to-Speech - Provides multilingual text-to-speech synthesis with voice cloning and streaming capabilities.
- Voice Cloning - Replicates a speaker's voice from a short reference audio sample for personalized speech generation.
- Streaming Audio Generators - Ships a streaming audio generation pipeline that outputs results incrementally for low-latency playback.
- Word-Level Timestamps - Provides word-level timestamp alignment for transcribed speech, mapping each word to precise audio positions.
- Speaker Diarizers - Detects speech segments and distinguishes between different speakers in audio recordings.
- Model Quantization - Reduces model weight precision to decrease size and accelerate inference for audio processing models.
- Weight Quantization - Loads model weights in reduced-precision formats at runtime to decrease memory footprint and accelerate inference.
- Speech Enhancers - Cleans noisy audio recordings and distinguishes between different speakers for clearer speech output.
- Voice Activity Detectors - Detects speech segments and distinguishes between different speakers in audio recordings.
- Audio Streaming Pipelines - Processes audio in chunks through a chain of models for real-time generation and transcription.
- Audio API Servers - Provides an interactive web UI and OpenAI-compatible REST API for audio generation and transcription.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
