# whisperspeech/whisperspeech

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/whisperspeech-whisperspeech).**

_How this analysis was created: the description and tags below were written by an AI model that read this project's README and public documentation pages; stars, license and language come straight from the GitHub API. The model does not read the source code._

4,617 stars · 271 forks · Jupyter Notebook · MIT

## Links

- GitHub: https://github.com/WhisperSpeech/WhisperSpeech
- Homepage: https://whisperspeech.github.io/WhisperSpeech/
- awesome-repositories: https://awesome-repositories.com/repository/whisperspeech-whisperspeech.md

## Topics

`pytorch` `speech-synthesis` `tts`

## Description

WhisperSpeech is a multilingual speech synthesizer and neural text-to-speech system. It functions by inverting the Whisper model architecture to convert text into high-fidelity synthetic audio.

The system enables voice cloning by using reference audio files to mimic specific speakers. It supports multilingual speech production, which includes the ability to generate audio across different languages and handle language switching within a single sentence.

The project covers a broad range of speech capabilities, including text-to-speech generation and speech dataset preparation. It incorporates tools for transcribing speech to text, extracting acoustic tokens, and detecting voice activity.

## Tags

### Artificial Intelligence & ML

- [Text-to-Speech](https://awesome-repositories.com/f/artificial-intelligence-ml/text-to-speech.md) — Generates high-fidelity synthetic audio from text using a neural multi-stage token pipeline. ([source](https://cdn.jsdelivr.net/gh/whisperspeech/whisperspeech@main/README.md))
- [Voice Cloning Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/speech-synthesis/voice-cloning-tools.md) — Enables mimicking specific speakers using reference audio files to guide synthetic speech generation. ([source](https://cdn.jsdelivr.net/gh/whisperspeech/whisperspeech@main/README.md))
- [Zero-Shot Voice Cloning](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/speech-synthesis/zero-shot-voice-cloning.md) — Provides zero-shot voice cloning by extracting speaker embeddings from short reference audio clips.
- [Multilingual Text-to-Speech Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/speech-and-text-conversion/text-to-speech-conversions/multilingual-text-to-speech-engines.md) — Specializes in multilingual speech production with seamless mixing of multiple languages in one output.
- [Voice Cloning Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/speech-synthesis-models/voice-cloning-engines.md) — Mimics specific human voices by using reference audio samples to guide the synthesis engine.
- [Inverted Architecture Models](https://awesome-repositories.com/f/artificial-intelligence-ml/text-to-speech-models/inverted-architecture-models.md) — Utilizes an inverted Whisper model architecture to convert text into high-fidelity synthetic audio.
- [Multilingual Speech Synthesizers](https://awesome-repositories.com/f/artificial-intelligence-ml/text-to-speech/cli-speech-synthesizers/multilingual-speech-synthesizers.md) — Produces spoken audio in multiple languages with the ability to switch languages within a single sentence. ([source](https://cdn.jsdelivr.net/gh/whisperspeech/whisperspeech@main/README.md))
- [Cross-Lingual Semantic Mappings](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/model-fine-tuning-adaptation/language-model-training/cross-lingual-objectives/cross-lingual-semantic-mappings.md) — Implements a shared semantic space to enable seamless language switching within a single synthetic speech stream.
- [Acoustic Token Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/natural-language-processing/text-tokenization/tokenization-pipelines/acoustic-token-pipelines.md) — Ships a multi-stage pipeline that separates linguistic and sonic features via semantic and acoustic tokenization.
- [Speech Dataset Engineering](https://awesome-repositories.com/f/artificial-intelligence-ml/speech-dataset-engineering.md) — Provides a complete pipeline for transcribing audio and extracting tokens to build speech synthesis training sets.
- [Speech-to-Text Transcribers](https://awesome-repositories.com/f/artificial-intelligence-ml/speech-to-text-transcribers.md) — Transcribes audio segments into text and semantic tokens to create high-quality training datasets. ([source](https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html))

### Graphics & Multimedia

- [Neural Audio Pipelines](https://awesome-repositories.com/f/graphics-multimedia/media-processing-analysis/audio-processing-systems/audio-processing-frameworks/neural-audio-pipelines.md) — Implements an end-to-end neural pipeline using semantic and acoustic tokens to generate high-fidelity synthetic speech.
- [Discrete Token Extraction](https://awesome-repositories.com/f/graphics-multimedia/media-processing-analysis/media-manipulation/media-processing-workflows/audio-analysis-synthesis/audio-feature-extraction/discrete-token-extraction.md) — Uses discrete token extraction to represent audio as quantized sequences for language-model-based speech generation.