AIGC-Audio/AudioGPT
AudioGPT
AudioGPT is an LLM-driven audio framework and processing suite that uses large language models to orchestrate neural audio pipelines. It functions as a multimodal audio generator and processing system, integrating a collection of pretrained models to handle speech synthesis, sound generation, and audio manipulation.
The system is distinguished by its ability to generate audio from diverse inputs, including text and images, and its capacity to produce synchronized talking head videos. It also operates as a neural speech translator, converting spoken language between different tongues while preserving meaning.
The project covers a broad range of audio capabilities, including restoration, source separation, and automatic speech transcription. Additional functional areas include sound analysis for event detection, spatial audio conversion from mono to binaural formats, and speech style transfer.
Features
- LLM Orchestrators - Uses a large language model to orchestrate the selection and workflow of specialized audio tools.
- LLM-Driven Orchestration - Uses large language models to orchestrate neural audio pipelines for generation and processing tasks.
- AI Audio Enhancement - Provides AI-driven enhancement to remove background noise and improve overall speech clarity.
- Multimodal Generation - Implements a system that generates soundscapes and audio clips from visual images or natural language descriptions.
- Audio Processing - Provides a comprehensive suite for speech transcription, source separation, and audio quality enhancement.
- Source Separation Tools - Isolates individual speakers or distinct sound sources from mixed audio signals into separate tracks.
- Audio Transcription - Converts spoken audio into written text to enable searchable analysis of sound files.
- Automatic Speech Recognition - Implements automatic speech recognition to transform spoken language into written text.
- Speech-to-Speech Translation - Converts spoken language between different tongues while preserving original meaning and tone.
- Speech Transcription - Provides automated conversion of spoken audio into written text for transcription and analysis.
- Text-to-Speech - Converts written text into natural-sounding spoken audio using neural text-to-speech models.
- Latent Acoustic Mapping - Maps visual and textual data to latent acoustic spaces to generate corresponding audio and soundscapes.
- Neural Audio Pipelines - Implements an end-to-end modular pipeline that integrates multiple pretrained neural models for audio tasks.
- Image-to-Audio Synthesis - Produces soundscapes or audio clips based on the visual content and context of an image.
- Audio Source Separation Models - Provides models for isolating specific speakers or sounds from mixed recordings into distinct tracks.
- Talking Head Generators - Generates animated talking head videos with facial movements synchronized to provided audio tracks.
- Agentic Goal Decomposition - Employs LLMs to recursively decompose complex audio requests into sequences of actionable sub-tasks.
- Audio Processing Frameworks - Provides a framework for orchestrating complex neural-based audio processing pipelines with asynchronous execution.
- Prompt-Driven Parameter Synthesis - Translates natural language descriptions into structured control signals to parameterize audio generation models.
- Singing Voice Synthesis - Generates melodic singing audio based on provided text and musical timing prompts.
- Audio Generation - Framework for understanding and generating speech, music, and sound.
- Large Language Models - Framework for understanding and generating speech, music, and sound.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
- Perception Models - Understanding and generating speech, music, and sound.
- Audio and Video Tools - Tool for understanding and generating speech, music, and sound effects.
- Chatbots and Assistants - Multimodal assistant for speech, music, and sound.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
