2noise/ChatTTS
ChatTTS
ChatTTS is a conversational text-to-speech generative model designed to convert written dialogue into natural sounding audio. It functions as a multilingual speech synthesis framework capable of producing human-like audio across different languages and speaker profiles.
The system is distinguished by its ability to generate interactive dialogue with realistic vocal nuances. It utilizes a speech nuance controller to insert specific tokens that trigger non-verbal elements, such as laughter, pauses, and interjections, during the synthesis process.
The project includes a streaming audio generator that delivers speech incrementally to reduce latency. It further supports multi-speaker embeddings to maintain consistent vocal characteristics throughout a conversation.
Features
- Conversational Audio Streams - Provides a generative model for natural, multi-speaker interactive dialogue and conversational audio streams.
- Audio Tokenization - Converts raw audio waveforms into discrete numerical codes for processing by the language model.
- Autoregressive Transformers - Implements an autoregressive transformer architecture to predict audio tokens for sequential speech generation.
- Prosody Control Tokens - Inserts specialized control tokens to trigger non-verbal vocal behaviors like laughter and pauses.
- Speaker Embeddings - Uses learned vector representations to maintain consistent vocal characteristics across different speakers.
- Multilingual Synthesis - Provides a generative framework capable of producing human-like speech across multiple languages.
- Text-to-Speech - Synthesizes natural human speech from written dialogue with human-like rhythms and nuances.
- Latent Acoustic Mapping - Maps natural language input to a latent space to guide the generation of acoustic features.
- Spoken Dialogue Generation - Creates spoken audio for multi-speaker interactions including realistic pauses and emotional cues.
- Vocal - Refines audio output by adding human-like non-verbal elements such as laughter and interjections.
- Speech Synthesis Markup Controls - Uses markup-like tokens to control prosody and insert fine-grained vocal elements like laughter.
- Vocal Nuance Controllers - Implements a controller that inserts vocal tokens to trigger specific audio elements like laughter.
- Chunked Audio Streaming - Streams audio output in chunks incrementally to minimize latency during speech generation.
- Generative Audio Chunking - Yields audio waveform chunks sequentially during generation to enable immediate low-latency playback.
- Additional AI Tools - Generative TTS model optimized for natural, expressive daily dialogue with fine-grained prosody control.
- Core Models - The official repository for the core model implementation.
- Speech Processing - Conversational text-to-speech for dialogue.
- Speech Synthesis - Conversational text-to-speech for dialogue.
