# heartmula/heartlib

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/heartmula-heartlib).**

3,921 stars · 298 forks · Python · apache-2.0

## Links

- GitHub: https://github.com/HeartMuLa/heartlib
- awesome-repositories: https://awesome-repositories.com/repository/heartmula-heartlib.md

## Description

Heartlib is an audio processing library for large language models that provides tools for audio tokenization, compression, and cross-modal alignment. It implements core models for audio-text embedding, automatic speech recognition, neural codecs, and text-driven audio synthesis.

The project features a text-to-audio synthesis engine capable of generating high-fidelity music and speech from text descriptions or reference files. It also includes a neural audio codec designed for low-bitrate compression that preserves acoustic structure and sound quality.

Additional capabilities cover audio-text alignment via a shared latent space for retrieval, as well as transcription tools specifically designed to convert vocal lyrics and singing into written text.

## Tags

### Part of an Awesome List

- [Audio Processing Libraries](https://awesome-repositories.com/f/awesome-lists/devtools/audio-processing-libraries.md) — A comprehensive library for audio tokenization, compression, and cross-modal alignment for large language models.
- [Cross-Modal Audio-Text Alignment](https://awesome-repositories.com/f/awesome-lists/ai/cross-modal-models/cross-modal-audio-text-alignment.md) — Maps music descriptions and audio segments into a shared space to retrieve matching files. ([source](https://github.com/HeartMuLa/heartlib#readme))

### Artificial Intelligence & ML

- [Audio Tokenization](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-tokenization.md) — Converts raw audio signals into discrete numerical representations to preserve acoustic structural patterns. ([source](https://github.com/HeartMuLa/heartlib/blob/main/README.md))
- [Low-Frame-Rate Tokenizers](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-tokenization/low-frame-rate-tokenizers.md) — Transforms continuous audio streams into discrete, low-frame-rate tokens for efficient data modeling.
- [Neural Audio Compression](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-tokenization/neural-audio-compression.md) — Uses neural tokenizers to achieve high-fidelity audio compression at low bitrates. ([source](https://github.com/HeartMuLa/heartlib#readme))
- [Audio Transcription](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-transcription.md) — Extracts spoken words from music files to convert vocal singing into written text. ([source](https://github.com/HeartMuLa/heartlib#readme))
- [Automatic Speech Recognition](https://awesome-repositories.com/f/artificial-intelligence-ml/automatic-speech-recognition.md) — Provides a pipeline to transcribe vocal lyrics and singing from audio signals into written text.
- [Text-to-Audio Synthesis](https://awesome-repositories.com/f/artificial-intelligence-ml/text-to-audio-synthesis.md) — Generates high-fidelity audio waveforms from text descriptions or latent representations using a neural synthesis engine.
- [Text-to-Music Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-music-composition/text-to-music-engines.md) — Synthesizes high-fidelity audio tracks based on written lyrics, style descriptions, or reference audio. ([source](https://github.com/HeartMuLa/heartlib#readme))
- [Audio-Text Embedding Models](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-text-embedding-models.md) — Implements a shared latent space for mapping music descriptions to audio segments.
- [Autoencoders](https://awesome-repositories.com/f/artificial-intelligence-ml/autoencoders.md) — Implements autoencoder architectures to compress audio data into latent representations for high-quality reconstruction.
- [Cross-Attention Mechanisms](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/generative-ai-architectures/cross-attention-mechanisms.md) — Implements cross-attention mechanisms to align textual descriptions with corresponding audio segments.
- [Shared Latent Spaces](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/generative-ai-models/latent-space-generative-models/shared-latent-spaces.md) — Maps audio and text into a unified shared latent space to enable cross-modal retrieval.

### Graphics & Multimedia

- [Cross-Modal Retrieval Alignment](https://awesome-repositories.com/f/graphics-multimedia/media-processing-analysis/audio-processing-systems/audio-processing/text-to-speech-engines/text-to-speech-engines/cross-modal-retrieval-alignment.md) — Maps music descriptions and audio segments into a shared space for cross-modal retrieval.