Awesome GitHub RepositoriesPhoneme-Based Alignment

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

CosyVoice is a speech synthesis framework that utilizes large language models to generate expressive, multilingual audio. The system functions as an audio generation engine capable of producing natural-sounding speech across multiple languages while preserving regional dialects and specific emotional tones. The platform distinguishes itself through its zero-shot voice cloning capabilities, which allow for the creation of synthetic voice profiles from short audio samples without requiring additional model training. It provides fine-grained control over vocal attributes, enabling users to adjus

WhisperX is an automated speech recognition toolkit designed to convert spoken audio into text while maintaining precise synchronization with the original media. It functions as an integrated pipeline that combines transcription, phoneme-based alignment, and speaker diarization to produce structured, attributed transcripts. The project distinguishes itself through its use of forced alignment, which matches existing text to audio signals at the phoneme level to generate accurate word-level timestamps. It also incorporates speaker diarization to identify and label unique voices within a recordi

Index-tts is a neural audio generation engine designed to convert written text into high-fidelity human speech. By utilizing deep learning models and phoneme-based sequence modeling, the system transforms text into natural-sounding audio waveforms suitable for a variety of accessibility and media applications. The platform functions as a server-side inference pipeline that provides a programmatic interface for integrating voice generation into external applications. It distinguishes itself through asynchronous audio streaming, which buffers and delivers generated speech chunks in real time to

Piper is a local neural text-to-speech engine designed to convert written text into natural human speech entirely on your own hardware. By utilizing a neural synthesis framework, it operates without the need for internet connectivity, ensuring that all audio generation remains private and secure. The system distinguishes itself through a modular architecture that allows for the dynamic loading of speaker embeddings and voice configurations. This enables users to switch between various vocal personas and styles without requiring a full reload of the core synthesis model. By processing input th

VoiceCraft is a neural speech generation and manipulation system consisting of a text-to-speech system, a voice cloning tool, and an audio inpainting engine. It uses a large language model approach to synthesize high-fidelity audio from text and replicate speaker identities. The system provides zero-shot voice cloning and speech editing capabilities, allowing users to modify spoken content within existing recordings. This includes an audio inpainting engine that replaces specific sections of audio with new speech while preserving the original acoustic characteristics and speaker identity. Th

EmotiVoice is an emotional text-to-speech engine and bilingual speech synthesizer designed to generate synthetic audio in English and Chinese. It utilizes a deep learning architecture to produce high-fidelity speech with controllable emotional states and timbres. The project includes a voice cloning framework for replicating specific speaker identities by training custom acoustic models on personal audio datasets. It employs a jointly-trained acoustic-vocoder pipeline and style-embedding-based synthesis to manage expression and reduce audio artifacts. The system covers a broad range of speec

YuE: Open Full-song Music Generation Foundation Model, something similar to Suno.ai but open

Muzic est une plateforme et un framework de deep learning pour l'analyse, la composition et la synthèse musicale assistées par IA. Il fonctionne comme un framework de génération musicale et un outil d'analyse, utilisant des modèles de langage étendus et des agents autonomes pour orchestrer la création et l'interprétation de musique symbolique et audio. Le projet se distingue par ses capacités intermodales, mappant le langage naturel et la musique symbolique dans un espace d'intégration commun pour la classification zero-shot et la recherche d'informations. Il emploie une variété d'architectures spécialisées, notamment des frameworks de diffusion pour la synthèse audio, des mécanismes d'attention à double grain pour la cohérence structurelle des séquences longues, et un système hybride qui combine les règles de théorie musicale avec des réseaux de neurones. La plateforme couvre un large éventail de capacités, y compris la génération de séquences MIDI à partir de texte et de paroles, la synthèse vocale neuronale et la transcription automatisée de paroles. Elle fournit également des outils pour la modélisation de la structure musicale, la génération symbolique basée sur des attributs et l'orchestration d'outils musicaux externes via des agents autonomes. Les utilitaires de support incluent des pipelines d'ingénierie de données pour la binarisation MIDI à grande échelle, l'encodage de jeux de données et le traitement du signal audio pour l'extraction de notes de mélodie et l'alignement parole-phonème.

Ce projet est un framework de conversion vocale TensorFlow et une boîte à outils audio de deep learning conçue pour le transfert de style vocal neuronal. Il fonctionne comme un moteur de synthèse vocale qui transforme les caractéristiques spectrales de la voix d'un locuteur source pour correspondre à l'identité vocale d'un locuteur cible. Le système emploie une approche basée sur les phonèmes pour la conversion vocale, classant les énoncés audio en phonèmes indépendants du locuteur et les resynthétisant en utilisant une voix cible. Ce pipeline permet la transformation des caractéristiques vocales en mappant les caractéristiques audio entre différents locuteurs. La boîte à outils inclut des capacités pour l'entraînement de modèles audio sur plusieurs GPU, la normalisation des données de tenseur et la gestion des hyperparamètres du modèle. Elle fournit également des outils pour surveiller les performances, comme la visualisation de la précision de classification via des matrices de confusion.

Voicevox is a text-to-speech synthesis software and audio production environment that converts written text into spoken audio using synthetic character voices. It functions as both a comprehensive editor for voice design and a standalone speech synthesis engine capable of generating audio via an API for integration into external applications. The project distinguishes itself by providing a singing voice synthesizer that uses a piano-roll interface for melodic vocal composition, including the ability to generate humming. It offers specialized prosody editing tools for the manual refinement of