yl4579/StyleTTS2
StyleTTS2
StyleTTS2 is an adversarial text-to-speech model that uses style diffusion and large speech language models to generate natural-sounding speech from text input. It combines adversarial training with large pre-trained speech models to improve speech quality and reduce artifacts, while employing a style diffusion process that extracts prosodic and timbral features from reference audio to guide speech generation.
The model supports multi-speaker voice synthesis by conditioning the diffusion process on speaker-specific embeddings derived from reference utterances, enabling voice cloning and adaptation to new speakers with minimal data. It offers style-controllable text-to-speech through diffusion-based sampling, and can generate speech directly from text without requiring a reference audio sample at inference time. The system uses a two-stage training pipeline that first trains on aligned data, then fine-tunes with unaligned data using style diffusion and adversarial loss.
Pre-trained model checkpoints are available for loading and running inference, with provided notebooks and importable scripts for generating speech from text. The model can be fine-tuned on a new speaker using a small amount of speech data, or trained from scratch on single or multiple voices using speaker labels and adjustable settings.
Features
- Text-to-Speech - Generating natural-sounding speech from text input without needing a reference audio sample.
- Text-To-Speech Models - An adversarial text-to-speech model that uses style diffusion and large speech language models to generate natural-sounding speech from text input.
- Text-to-Speech Synthesizers - Converts text into natural-sounding synthetic speech using adversarial training and style diffusion.
- TTS Adversarial Frameworks - Improving speech quality and naturalness through adversarial training with large speech language models.
- Style Encoders - Extracts prosodic and timbral style features from a short reference audio clip to guide the diffusion-based speech generation.
- Adversarial Speech Training - Uses a discriminator trained on representations from a large pre-trained speech model to improve naturalness and reduce artifacts.
- Two-Stage Training Pipelines - First trains a text-to-speech model on aligned data, then fine-tunes with unaligned data using style diffusion and adversarial loss.
- Multi-Speaker Training - Supports training on multiple voices by using speaker labels to sample reference audio for style diffusion during training.
- Speaker Embeddings - Controls voice identity by conditioning the diffusion process on speaker-specific embeddings derived from reference utterances.
- Reference-Free Style Controls - Controls speaking style through diffusion-based sampling without requiring a reference audio sample at inference time.
- Adversarial Speech Generators - Employs adversarial training with large speech language models to improve speech quality and naturalness.
- Diffusion-Based Speech Style Transfers - Producing expressive speech with varied styles by leveraging diffusion-based style transfer from reference audio.
- Style-Conditioned Diffusion Decoders - Generates speech by iteratively denoising a latent representation conditioned on style embeddings extracted from reference audio.
- Pre-trained Weight Loading - Loads a pre-trained model checkpoint and generates speech from text without requiring a reference audio sample.
- Speech Model Fine-Tuning - Adapts a pre-trained multi-speaker model to a new speaker using a small amount of speech data for reduced training time.
- Few-Shot Voice Cloning - Adapting a pre-trained model to a new speaker using a small amount of speech data for personalized voice generation.
- Speaker Adaptation - Adapt a pre-trained multi-speaker model to a new speaker using a small amount of speech data to reduce training time.
- Multi-Speaker Synthesis - Generates speech in multiple voices by sampling reference audio for style diffusion during training and inference.
- Inference Scripts - Provides notebooks and scripts to load pre-trained speech models and generate speech from text.
