HeartMuLa/heartlib

Features

• Audio Processing Libraries - A comprehensive library for audio tokenization, compression, and cross-modal alignment for large language models.
• Audio Tokenization - Converts raw audio signals into discrete numerical representations to preserve acoustic structural patterns.
• Low-Frame-Rate Tokenizers - Transforms continuous audio streams into discrete, low-frame-rate tokens for efficient data modeling.
• Neural Audio Compression - Uses neural tokenizers to achieve high-fidelity audio compression at low bitrates.
• Audio Transcription - Extracts spoken words from music files to convert vocal singing into written text.
• Automatic Speech Recognition - Provides a pipeline to transcribe vocal lyrics and singing from audio signals into written text.
• Text-to-Audio Synthesis - Generates high-fidelity audio waveforms from text descriptions or latent representations using a neural synthesis engine.
• Text-to-Music Engines - Synthesizes high-fidelity audio tracks based on written lyrics, style descriptions, or reference audio.
• Audio-Text Embedding Models - Implements a shared latent space for mapping music descriptions to audio segments.
• Autoencoders - Implements autoencoder architectures to compress audio data into latent representations for high-quality reconstruction.
• Cross-Attention Mechanisms - Implements cross-attention mechanisms to align textual descriptions with corresponding audio segments.
• Shared Latent Spaces - Maps audio and text into a unified shared latent space to enable cross-modal retrieval.
• Cross-Modal Audio-Text Alignment - Maps music descriptions and audio segments into a shared space to retrieve matching files.
• Cross-Modal Retrieval Alignment - Maps music descriptions and audio segments into a shared space for cross-modal retrieval.

Star history