Voice and audio processing

Kaldi is an automatic speech recognition toolkit used to train and deploy models that convert spoken audio into text. It functions as a framework for designing and evaluating acoustic and language models through a structured pipeline of processing tools. The system acts as a cross-platform speech engine, capable of compiling recognition logic for Android and WebAssembly to enable execution on mobile devices and web browsers. It also includes a dedicated converter for migrating speech recognition models from the HTK format into a compatible internal structure. The toolkit covers a broad range

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

This project is a speech recognition and translation engine that utilizes a sequence-to-sequence transformer architecture to convert audio into text. It is built upon a weakly supervised learning framework, which leverages large-scale, unlabelled audio-transcript data to create generalized speech representations capable of performing simultaneous transcription, language identification, and translation. The system distinguishes itself through a unified multi-task modeling approach that shares token sequences across different objectives, allowing it to handle diverse languages and vocabularies

PaddleSpeech is a comprehensive toolkit of neural models for speech recognition, synthesis, and translation built on the PaddlePaddle deep learning framework. It provides a collection of frameworks and tools for converting spoken audio into written text, synthesizing natural audio from text, and performing direct speech translation. The toolkit includes specialized capabilities for keyword spotting to detect trigger words and speaker verification systems that extract unique voiceprints to identify and distinguish between individuals. It also features end-to-end translation tools that map audi

Sherpa-ONNX is an ONNX-based speech processing toolkit that provides a local speech recognition engine, an on-device voice synthesis tool, and a speaker identification framework. It is designed as a cross-platform speech API that enables speech-to-text, text-to-speech, and speaker verification tasks to be executed locally on a device without requiring network access. The project is distinguished by its ability to perform zero-shot voice cloning and speaker diarization on-device. It supports a wide range of hardware accelerations, including GPU and various NPU architectures, and provides a Web

ESPnet is a comprehensive speech processing toolkit and PyTorch-based trainer designed for building end-to-end speech recognition, synthesis, and translation models. It provides a structured framework for developing automatic speech recognition systems using transducer and encoder-decoder architectures, alongside engines for text-to-speech synthesis and speech translation pipelines. The project distinguishes itself through a recipe-based workflow execution system that ensures experimental reproducibility by running standardized sequences of scripts for data preparation and model training. It

This library provides a deep learning framework for training neural networks to perform speech recognition and audio classification. It utilizes sequence-to-sequence architectures to map variable-length audio inputs into text or numerical outputs, enabling the development of custom speech-to-text transcription models. The project distinguishes itself through integrated audio processing capabilities that transform raw waveforms into spectrograms and high-dimensional numerical vectors. These tools allow for the extraction of unique vocal characteristics to identify speakers, as well as the clas

AudioKit is an audio framework for iOS, macOS, and tvOS that provides tools for digital audio synthesis, signal processing, and audio analysis. It functions as a synthesis engine for generating audio waveforms and textures, a processing library for modifying tonal characteristics, and a toolkit for extracting frequency and amplitude data from sonic signals. The framework utilizes a modular node architecture and graph-based signal routing to connect audio generators, processors, and outputs. It wraps low-level audio primitives in high-level classes to facilitate sound generation and modificati

Vosk is an offline speech-to-text engine and API that converts spoken audio into text locally on a device. It provides a cross-platform speech toolkit with language bindings for integrating voice recognition into server environments, Android, iOS, and Raspberry Pi. The project includes a speaker identification tool to distinguish between different voices and an acoustic model trainer for building custom neural network models. These training tools enable speech feature extraction and model accuracy evaluation to improve recognition for specialized domains. The system supports real-time audio

wav2letter is an automatic speech recognition toolkit and deep learning framework designed to convert audio speech signals into written text. It functions as a distributed training system and an inference engine for building and deploying neural network architectures. The system enables the training of large-scale speech models across multiple compute nodes using custom architecture files and structured recipes. It includes an inference engine that allows these trained models to be executed within Python workflows to transform audio sequences into text. The framework covers the full speech r

DeepSpeech is an open-source speech-to-text framework and machine learning engine designed to convert spoken audio into written text locally on a device. It provides on-device speech recognition that operates without requiring an internet connection to external servers. The system supports real-time speech transcription across a variety of hardware platforms, ranging from single-board computers and edge devices to GPU servers. This allows for audio analysis and processing directly on the local hardware.

AudioGPT is an LLM-driven audio framework and processing suite that uses large language models to orchestrate neural audio pipelines. It functions as a multimodal audio generator and processing system, integrating a collection of pretrained models to handle speech synthesis, sound generation, and audio manipulation. The system is distinguished by its ability to generate audio from diverse inputs, including text and images, and its capacity to produce synchronized talking head videos. It also operates as a neural speech translator, converting spoken language between different tongues while pre

Whisper.cpp is a high-performance, local-first speech recognition engine designed to run large-scale machine learning models on consumer hardware. It functions as a portable library that converts audio into text, supporting both static file transcription and real-time stream processing. By utilizing a lightweight inference engine and weight quantization, the project minimizes memory and compute overhead, allowing for efficient execution without reliance on external cloud APIs or internet connectivity. The project distinguishes itself through a hardware-agnostic compute abstraction that offloa

Librosa is a Python audio analysis library and digital signal processing framework. It functions as a feature extraction suite and music information retrieval tool designed to analyze the structural and sonic characteristics of audio signals. The library provides specialized capabilities for music analysis, including dynamic tempo tracking to identify rhythmic pulses and spectral feature extraction to compute harmonic spectra, chroma variants, and onset points. It also serves as a time-series audio processor for synchronizing audio streams. The system covers a broad range of audio processing

Tortoise-tts is a neural text-to-speech engine and voice cloning toolkit designed for high-quality audio generation. It functions as a zero-shot synthesis system, meaning it can generate speech for unseen speakers without requiring additional training or fine-tuning for each new voice. The system specializes in replicating human vocal characteristics using small sets of reference audio clips. It allows for the extraction of voice latents to mimic specific speakers, the generation of random synthetic identities, and the blending of multiple voice profiles to create hybrid vocal identities. Th

This project is a deep learning text-to-speech toolkit used for training and deploying neural speech synthesis models. It provides a comprehensive framework for converting written text into spoken audio, utilizing neural vocoders to transform synthesized spectrograms into high-fidelity audio waveforms. The toolkit includes a voice cloning system that replicates specific human voices by extracting speaker embeddings from short audio samples. It also supports multi-speaker audio synthesis, allowing the generation of speech across different vocal identities using specialized model architectures.

Aubio is an audio analysis and digital signal processing library designed for music information retrieval. It provides a suite of tools for extracting musical features, estimating fundamental frequencies, and tracking rhythmic pulses in audio streams. The library specializes in the detection of pitch and beat, enabling the extraction of musical notes and the estimation of overall tempo. It also includes capabilities for automatic onset detection to identify the start of sonic events and the separation of audio signals into percussive transients and steady-state tonal components. The system c

F5-TTS is a text-to-speech system that utilizes a flow matching engine and diffusion transformers to generate fluent synthetic speech. It functions as a multilingual speech synthesizer and neural training framework, providing tools for voice cloning and high-performance inference serving. The project distinguishes itself through a voice cloning toolkit capable of mimicking specific speaker characteristics and tones from reference audio clips. It supports cross-lingual generation, allowing for the synthesis of audio across various global languages or the mixing of multiple languages within a s

Chatterbox is a comprehensive machine learning platform designed for multilingual speech synthesis and real-time audio generation. It functions as an engine that converts text into natural-sounding speech, capable of replicating specific human vocal characteristics and emotional expressions from short audio samples. The platform distinguishes itself through advanced control over the synthesis process, allowing for the manipulation of emotional intensity and the injection of non-verbal vocalizations such as laughter or coughing. It is engineered for low-latency performance, utilizing an optimi

GPT-SoVITS is a text-to-speech synthesis engine and voice cloning toolkit designed for generating natural-sounding human speech. It functions as a neural audio processing pipeline that maps input text to high-fidelity audio waveforms, utilizing conditional variational autoencoders and flow-based decoders to ensure expressive output. The platform distinguishes itself through its ability to perform few-shot voice cloning and cross-lingual speech generation, allowing users to maintain a specific speaker's vocal identity and emotional delivery across multiple languages. By employing cross-modal l

This project is a comprehensive toolkit for on-device speech recognition, synthesis, and audio processing, specifically engineered for Apple Silicon. It provides a framework for building real-time, full-duplex voice agents that operate entirely offline, leveraging native hardware acceleration to maintain performance and privacy. By utilizing optimized machine learning models, the library enables local execution of complex audio tasks without reliance on external cloud services. The library distinguishes itself through its specialized focus on local, high-performance voice interaction. It incl

mlx-audio is an audio processing toolkit built on Apple MLX that provides speech transcription, text-to-speech synthesis, voice cloning, and audio source separation using local models. It offers an OpenAI-compatible REST API and web interface for running audio generation and transcription tasks, enabling drop-in integration with existing tools that follow that endpoint structure. The toolkit supports text-prompted audio source separation, allowing specific sounds to be isolated from mixed recordings based on natural language descriptions. It also provides voice cloning from a short reference

Rhubarb is an automated lip sync generator and phonetic speech analyzer that converts audio recordings into timed mouth-shape animation data. It identifies sounds and syllables within audio files to map them to specific visual mouth shapes, serving as an animation timing exporter for external character animation software. The tool utilizes a language-independent phonetic recognizer to process speech regardless of the spoken language. To increase accuracy, it supports dialogue-guided recognition by using external text files to guide the phonetic analysis of specific spoken scripts. The system

Amphion is an audio generation toolkit designed for the research and development of models that synthesize speech, music, and environmental sound effects. It provides a standardized framework for reproducible audio synthesis, incorporating a text-to-speech engine and a voice conversion framework. The project specializes in transforming audio identities, allowing for the modification of speaker accents and voice identities while preserving original rhythm and style. It also includes capabilities for singing voice synthesis and the generation of environmental soundscapes from text descriptions

Index-tts is a neural audio generation engine designed to convert written text into high-fidelity human speech. By utilizing deep learning models and phoneme-based sequence modeling, the system transforms text into natural-sounding audio waveforms suitable for a variety of accessibility and media applications. The platform functions as a server-side inference pipeline that provides a programmatic interface for integrating voice generation into external applications. It distinguishes itself through asynchronous audio streaming, which buffers and delivers generated speech chunks in real time to

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

This application is a platform for AI voice synthesis and neural voice cloning. It provides a comprehensive toolkit for converting text into natural-sounding human speech by applying custom-trained neural network models to specific audio samples. The system facilitates the entire lifecycle of voice model development, including the preparation of raw audiobooks and video transcriptions into structured training datasets. It supports the training of these models on local or remote hardware, utilizing multi-GPU distributed processing to handle large-scale data and accelerate model convergence. B

VibeVoice is a generative artificial intelligence platform designed for text-to-speech synthesis. It functions as a neural audio generation framework that converts written text into natural-sounding spoken audio, specifically engineered to maintain consistent vocal characteristics and narrative prosody across extended passages of content. The system distinguishes itself through its ability to generate long-form conversational speech while preserving speaker identity and linguistic content. By utilizing latent space disentanglement, the model separates speaker traits from the input text, allow

This project is a neural text-to-speech engine and voice cloning toolkit designed to generate synthetic speech that mimics the vocal characteristics of a target speaker. It functions as a real-time audio synthesizer, utilizing a deep learning pipeline to convert written text into high-fidelity speech output with minimal latency. The system employs a transfer learning framework that leverages pre-trained speaker verification models to adapt synthesis to new, unseen vocal identities. By using an encoder-based speaker embedding process, the toolkit maps variable-length audio samples into a laten

Magenta is a comprehensive toolkit for training, synthesizing, and performing music through neural models and hardware-integrated engines. It functions as a machine learning framework that enables the generation, manipulation, and real-time performance of audio, providing the structural foundations for musical intelligence through hierarchical sequence modeling and symbolic processing. The project distinguishes itself by enabling real-time, low-latency neural audio synthesis that can be integrated directly into professional digital audio workstations. It supports interactive musical jamming a

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

CosyVoice is a speech synthesis framework that utilizes large language models to generate expressive, multilingual audio. The system functions as an audio generation engine capable of producing natural-sounding speech across multiple languages while preserving regional dialects and specific emotional tones. The platform distinguishes itself through its zero-shot voice cloning capabilities, which allow for the creation of synthetic voice profiles from short audio samples without requiring additional model training. It provides fine-grained control over vocal attributes, enabling users to adjus

This software is a real-time voice changer that utilizes machine learning inference to transform live microphone input into target vocal characteristics. It functions as an artificial intelligence audio processing tool designed to modify vocal identity during active communication or live broadcasts. The application distinguishes itself by executing neural network models directly within the browser environment. It leverages web-based compute acceleration and dedicated audio threading to maintain low-latency performance, allowing users to switch between different voice profiles while processing

This project is a comprehensive software suite for voice synthesis and model management, providing a framework for training custom acoustic models and performing voice conversion. It utilizes deep-learning-based acoustic modeling to map source audio characteristics to target voice identities, enabling the transformation of input audio into specific vocal profiles. The system distinguishes itself through a feature-retrieval-based inference mechanism, which employs vector index files to perform nearest-neighbor searches on acoustic features for high-fidelity timbre matching. Users can manage th

Julius is a high-performance, open-source speech recognition engine designed for large vocabulary continuous speech recognition. It functions as a comprehensive framework utilizing Hidden Markov Model-based acoustic modeling and N-gram language models to convert live or recorded audio into text. The engine is built to support real-time streaming and provides a network-accessible service that allows external applications to manage recognition sessions and receive transcription results through programmatic commands. The engine distinguishes itself through its modular architecture and support fo

Pipecat is a framework and software development kit for building real-time multimodal AI agents and speech-to-speech systems. It utilizes a frame-based data pipeline to route audio, video, and text through a modular sequence of processors, enabling the orchestration of low-latency conversational AI. The project is distinguished by its ability to coordinate complex multimodal services, including speech-to-text, language models, and text-to-speech, within a single pipeline. It features semantic voice activity detection for natural turn-taking, state-machine conversation flows for dialogue manag

Bark is a generative audio engine and machine learning inference library designed to convert written text into high-fidelity speech and sound effects. It functions as a text-to-audio transformer, utilizing multi-stage neural network architectures to map semantic input tokens into detailed audio codebooks for synthesis. The system distinguishes itself through a hierarchical transformer stacking approach that separates semantic understanding from acoustic realization. By employing autoregressive token prediction and vector quantized codebook mapping, the engine bridges linguistic and sonic doma

This project is a comprehensive suite for neural speech synthesis, featuring a deep learning text-to-speech engine, a neural speech synthesis trainer, and a voice cloning toolkit. It provides a system for synthesizing human-like speech from text using neural network models and high-fidelity vocoders. The suite includes a speech model conversion utility to transform deep learning models between different formats for deployment across various hardware runtimes. It also provides a self-contained HTTP server to expose pre-trained text-to-speech models as a remote audio API. Capabilities include

PocketSphinx is an offline speech recognition engine that converts raw audio from files or live microphone streams into written text without requiring a network connection. It functions as a speech-to-text library, a real-time transcription engine, and a voice command processor, capable of detecting and transcribing spoken commands from continuous audio streams with configurable acoustic and language models. The engine uses weighted finite-state transducers to represent acoustic, phonetic, and language models as a single search graph for efficient decoding. It employs fixed-point acoustic mod

Data manipulation and transformation for audio signal processing, powered by PyTorch

Python speech features is a signal processing toolkit and library for extracting standard speech recognition features from raw audio signals. It provides computational capabilities to calculate mel-frequency cepstral coefficients, raw and log filterbank energies, and spectral subband centroids for automated speech recognition systems and acoustic analysis. The library implements audio signal transformations including pre-emphasis filtering, overlapping windowed frame segmentation, Fast Fourier Transform spectral analysis, mel-filterbank projection with configurable filter weights and frequenc

This project is a TensorFlow voice conversion framework and deep learning audio toolkit designed for neural voice style transfer. It functions as a speech synthesis engine that transforms the spectral characteristics of a source speaker's voice to match the vocal identity of a target speaker. The system employs a phoneme-based approach to voice conversion, classifying audio utterances into speaker-independent phonemes and resynthesizing them using a target voice. This pipeline allows for the transformation of voice characteristics by mapping audio features between different speakers. The too

This project is a GPU-accelerated speech engine and AI voice cloning tool. It functions as a text-to-speech synthesizer and voice-to-voice converter that replicates specific human voices to generate synthetic speech. The system creates digital voice profiles by analyzing short audio samples or capturing live microphone input. These profiles enable the transformation of existing audio recordings into a target speaker's voice or the synthesis of new audio from written text. The engine supports subtitle-based speech generation for batch processing and automated dubbing workflows. A web-based au

Zonos is a controllable audio synthesis engine and large language model for text-to-speech. It serves as a multilingual speech generator capable of producing audio in English, Japanese, Chinese, French, and German. The system provides zero-shot voice cloning, allowing the replication of specific human voices using short audio samples. It supports the capture of nuanced behaviors, such as whispering, and provides parametric control over speaking rate, pitch, frequency, and emotional tone. The project covers a broad range of expressive speech synthesis and custom audio generation capabilities,

NAudio is a .NET audio library that provides playback, recording, format conversion, and signal processing capabilities for Windows applications. It serves as an audio format converter between common formats like WAV, MP3, AAC, and AIFF using ACM or Media Foundation codecs, and functions as an audio playback and recording API through Windows audio APIs including WaveOut, DirectSound, ASIO, and WASAPI. The library includes an audio stream processor that can mix, trim, loop, fade, resample, and adjust pitch or channel count of audio streams with a floating-point engine. It also provides an audi

An audio server, programming language, and IDE for sound synthesis and algorithmic composition.