bytedance/MegaTTS3
MegaTTS3
MegaTTS3 is a bilingual speech synthesis system that generates natural-sounding speech in Chinese and English, including seamless code-switching within a single utterance. It functions as a text-to-speech engine, voice cloning system, and speech-to-text alignment tool, built around an acoustic latent compression model that encodes high-resolution audio into compact representations for efficient processing.
The system distinguishes itself through accent intensity control, allowing adjustment of a speaker's accent strength in generated speech, and voice cloning from short audio samples for personalized synthesis. It provides both a command-line interface for automated speech generation without a graphical environment and a web-based inference UI for browser-driven voice sample upload and text-to-speech output. A pseudo-label aligner trains text-speech alignment models using expert-generated labels for robust alignment.
Additional capabilities include grapheme-to-phoneme conversion for improved pronunciation accuracy, latent diffusion transformer-based audio reconstruction, and support for bilingual speech synthesis with code-switching. The system compresses speech into acoustic latents for efficient storage and downstream voice conversion tasks.
Features
- Bilingual Speech Synthesizers - Generates natural-sounding speech in Chinese and English, including code-switching within a single utterance.
- Text-to-Speech Engines - Converts written text into natural-sounding speech using a lightweight diffusion transformer model.
- Latent Space Encoders - Encodes high-quality audio into a compact latent representation that can be reconstructed with minimal loss.
- Zero-Shot Voice Cloning - Replicates a speaker's voice using only a brief audio reference for personalized speech synthesis.
- Grapheme To Phoneme Conversion - Converts written text into phonetic representations for improved pronunciation accuracy.
- Acoustic-Text Alignment - Aligns spoken audio to its corresponding text using a robust aligner trained on pseudo-labels from expert models.
- Text-to-Speech - Converts written text into natural-sounding speech using a lightweight diffusion transformer model.
- Latent Acoustic Mapping - Encodes high-resolution audio into a compact latent representation for efficient model training and voice conversion.
- Acoustic Latent Compressors - Encodes high-resolution audio into a compact latent representation for efficient model training and voice conversion.
- Command-Line Speech Synthesizers - Accepts a voice sample and text as arguments to produce speech output without a graphical interface.
- Speech Latent - Encodes speech into a compact latent space and reconstructs audio using a diffusion-based transformer decoder.
- Voice Cloning - Replicates a speaker's voice characteristics using only a brief audio reference, enabling personalized speech synthesis.
- Speech Accent Transformation - Provides accent intensity control by scaling learned accent embeddings during inference.
- Bilingual Code-Switching - Supports seamless switching between Chinese and English within a single utterance.
- Bilingual Synthesizers - Generates speech in Chinese and English, including code-switching within a single utterance.
- Speech-Text Pseudo-Label Aligners - Aligns spoken audio with its corresponding text transcription using a robust aligner trained on pseudo-labels from expert models.
- Command-Line Speech Synthesizers - Runs speech synthesis from a command line by providing a voice sample and text as arguments.
- Web-Based Speech Synthesizers - Provides a browser-based UI for uploading voice samples and generating speech from text.
- Web-Based Speech Inference UIs - Provides a browser interface for uploading voice samples and generating speech from text.
- Accent Intensity Controllers - Adjusts the strength of a speaker's accent in the generated speech through configurable weight parameters.
- Command Line Interfaces - Ships a command-line interface for generating speech from text and voice samples.
- Speech Processing - High-performance text-to-speech model.
- Speech Synthesis - High-performance speech synthesis model.
