SesameAILabs/csm
Csm
CSM is a conversational speech generation model and text-to-speech engine that converts text and audio inputs into synthetic speech. It utilizes a large language model architecture to predict and decode audio tokens for voice synthesis.
The system functions as a zero-shot voice cloner, replicating specific speaker identities using short audio samples without requiring additional training. This enables precise control over speaker identity and the creation of synthetic speech that mimics a specific person.
The model covers conversational speech synthesis and text-to-speech generation, transforming written text into spoken audio while maintaining natural flow and cadence.
Features
- Synthetic Speech Generation - Generates natural-sounding synthetic speech that replicates human conversational cadence and vocal characteristics.
- Audio Generation Models - Utilizes a large language model architecture to predict and decode audio tokens for voice synthesis.
- Waveform Decoders - Uses specialized neural waveform decoders to transform internal latent representations into audible speech.
- Voice Conditioning Encoders - Utilizes voice conditioning encoders to extract vocal identity from audio samples for speech synthesis.
- Speech Synthesis - Synthesizes realistic spoken dialogue that maintains natural conversational flow and cadence.
- Zero-Shot Voice Cloning - Implements zero-shot voice cloning to replicate speaker identities from short audio samples without additional training.
- Text-to-Speech - Transforms written text into high-quality spoken audio using neural processing and decoding.
- Text-to-Speech Engines - Provides a processing pipeline that transforms written text into high-fidelity spoken audio.
- Speaker Identity Control - Provides precise control over speaker identity using audio samples to mimic specific people.
- Multi-Modal Input Processors - Integrates diverse inputs, including text prompts and reference audio segments, into a shared representation.
- Sequence-to-Sequence Tasks - Employs sequence-to-sequence neural processing to map text and audio inputs to speech outputs.
- Vocal Characteristic Adjustments - Allows adjustment of vocal characteristics and tone to match a desired persona or individual.
- Latent Acoustic Mapping - Implements latent acoustic mapping to translate textual and acoustic cues into speech representations.
