Open-source alternatives to ACE Step

Muzic is a deep learning platform and framework for AI-driven music analysis, composition, and synthesis. It functions as a music generation framework and analysis tool, utilizing large language models and autonomous agents to orchestrate the creation and interpretation of symbolic and audio music. The project is distinguished by its cross-modal capabilities, mapping natural language and symbolic music into a shared joint embedding space for zero-shot classification and information retrieval. It employs a variety of specialized architectures, including diffusion frameworks for audio synthesis

DiffSinger is an AI vocal synthesizer and neural audio generator designed to produce high-fidelity singing and speech. It functions as a text-to-speech system and a diffusion-based singing voice synthesis tool that transforms text and pitch into audible audio. The system utilizes a shallow diffusion mechanism and iterative noise refinement to generate realistic vocal performances. It incorporates specialized sampling plugins and numerical solvers to accelerate inference and reduce the time required to generate synthetic voices. The project covers acoustic modeling, mel-spectrogram synthesis,

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

F5-TTS is a text-to-speech system that utilizes a flow matching engine and diffusion transformers to generate fluent synthetic speech. It functions as a multilingual speech synthesizer and neural training framework, providing tools for voice cloning and high-performance inference serving. The project distinguishes itself through a voice cloning toolkit capable of mimicking specific speaker characteristics and tones from reference audio clips. It supports cross-lingual generation, allowing for the synthesis of audio across various global languages or the mixing of multiple languages within a s

Mochi is an open-source text-to-video diffusion model designed to synthesize high-fidelity video sequences from natural language prompts. It utilizes a diffusion transformer architecture to generate temporal video data. The project includes a framework for low-rank adaptation, allowing the model to be fine-tuned on custom datasets to specialize visual styles or specific subjects. It also features a distributed inference engine that spreads model workloads across multiple graphics cards to increase memory capacity and processing speed. The system covers programmable video generation through a

Amphion is an audio generation toolkit designed for the research and development of models that synthesize speech, music, and environmental sound effects. It provides a standardized framework for reproducible audio synthesis, incorporating a text-to-speech engine and a voice conversion framework. The project specializes in transforming audio identities, allowing for the modification of speaker accents and voice identities while preserving original rhythm and style. It also includes capabilities for singing voice synthesis and the generation of environmental soundscapes from text descriptions

Audiocraft is a deep learning audio library and machine learning framework designed for training, fine-tuning, and evaluating generative models for music and sound effects. It functions as a text-to-music generative model and a neural audio codec, providing the tools necessary to compress audio signals into discrete representations and synthesize high-fidelity waveforms from textual descriptions. The framework is distinguished by its ability to combine multiple conditioning signals, allowing for the generation of audio based on text prompts, melodic excerpts, or style-based audio clips. It al

This repository provides a collection of reference implementations and code examples for training and deploying machine learning models using the MLX framework. It serves as a practical guide for executing distributed training, fine-tuning large language models, converting model weights, and implementing multimodal generative workflows. The project distinguishes itself through specialized examples for local hardware execution, featuring weight quantization to reduce memory usage and low-rank adaptation for parameter-efficient fine-tuning. It also includes scripts for transforming external mod

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

ace-step-ui is an AI music production workspace and interface for generating, editing, and organizing synthetic audio tracks and vocals. It provides a technical control panel for managing prompts, seeds, and style parameters to produce high-quality audio. The project includes a digital audio workstation interface for trimming and fading files, alongside an audio stem separation tool that splits mixed tracks into individual components such as drums, bass, and vocals. It also features a music video creator for generating visual content and procedural album art to accompany generated music. The

This is a PyTorch deep learning framework and tool for human motion synthesis that generates 3D character animations from text prompts or action descriptions. It functions as a text-to-motion generator that converts natural language and categorical labels into temporally consistent 3D skeletal movement sequences. The system utilizes a transformer-based diffusion model to iteratively denoise motion data. It includes capabilities for action-conditioned generation, monocular-to-3D motion lifting, and motion sequence editing using text constraints. The framework incorporates geometric motion con

YuE: Open Full-song Music Generation Foundation Model, something similar to Suno.ai but open

Civitai is a platform for generative media creation and AI model distribution. It provides a centralized service for producing images, videos, audio, and music, while serving as a repository where users can share, discover, and browse custom model weights and fine-tuned adaptations. The platform distinguishes itself through a provider-agnostic orchestration layer that manages multi-step generation pipelines and complex workflows across different backends. It integrates with autonomous AI agents and editors via the Model Context Protocol, allowing external tools to access generation pipelines

Voicevox is a text-to-speech synthesis software and audio production environment that converts written text into spoken audio using synthetic character voices. It functions as both a comprehensive editor for voice design and a standalone speech synthesis engine capable of generating audio via an API for integration into external applications. The project distinguishes itself by providing a singing voice synthesizer that uses a piano-roll interface for melodic vocal composition, including the ability to generate humming. It offers specialized prosody editing tools for the manual refinement of

DiT is a latent diffusion model and transformer-based generative AI framework implemented in PyTorch. It functions as a class-conditional image generator that replaces traditional convolutional backbones with a transformer architecture to synthesize high-fidelity images. The project utilizes patch-based latent processing and latent space compression to operate on low-dimensional image representations. It incorporates class-conditional guidance and adjustable guidance scales to control the visual content of generated images during the sampling process. The framework covers distributed model t

Open-Sora-Plan is a text-to-video framework and distributed video training system. It utilizes a diffusion transformer architecture and large language model components to transform written descriptions or image prompts into high-quality video sequences. The system features a distributed infrastructure designed for large-scale video training and inference. It employs sequence parallelism to split high-resolution or long-duration video samples across multiple GPUs and uses a sparse attention mechanism to increase processing speed. The project includes capabilities for both text-to-video and im

HunyuanDiT is a bilingual text-to-image generative model and diffusion transformer image generator. It uses a latent diffusion system to synthesize high-resolution images from text prompts, with a specific focus on understanding and generating content from both Chinese and English language descriptions. The project features a multi-resolution transformer architecture and a bilingual embedding space to map different scripts into a shared semantic area. It supports iterative multi-turn image refinement, which translates conversational dialogue into updated prompts to progressively modify visual

This project is an AI singing voice conversion system and vocal processor used for training generative voice models and converting vocal recordings or live input into a target voice. It functions as a VITS model trainer and a real-time voice changer that transforms vocal timbre and pitch to change the identity of a singer. The system provides a graphical management dashboard for controlling training hyperparameters and voice conversion presets. It supports low-latency audio streaming for live microphone input and employs pitch estimation to ensure precise matching between source and target vo

VoiceCraft is a neural speech generation and manipulation system consisting of a text-to-speech system, a voice cloning tool, and an audio inpainting engine. It uses a large language model approach to synthesize high-fidelity audio from text and replicate speaker identities. The system provides zero-shot voice cloning and speech editing capabilities, allowing users to modify spoken content within existing recordings. This includes an audio inpainting engine that replaces specific sections of audio with new speech while preserving the original acoustic characteristics and speaker identity. Th

This project is an AI voice training framework and singing voice conversion tool. It uses VITS and SoftVC technologies to transform the timbre of singing and spoken audio recordings, allowing a user to change the vocal characteristics of a recording to match a specific target speaker. The system provides a web-based voice converter interface for managing model checkpoints and performing timbre transformation and pitch shifting. It supports exporting trained models to the ONNX format for use in external interfaces and lightweight runtimes. The framework covers the full production pipeline, in

Open-Sora is a video generation framework designed to produce cinematic sequences from text prompts and images. It functions as a generative system that transforms written descriptions or reference images into video content featuring realistic textures and lighting. The project includes a dedicated prompt engineering tool that uses large language models to expand simple user inputs into detailed descriptions. It also features a motion controller for adjusting movement intensity in generated sequences and evaluating motion levels in existing video files. The framework incorporates text-to-vid

This project is a multimodal AI proxy and content generation hub that provides a unified web interface for interacting with multiple large language models and generative AI services. It functions as a secure API access gateway, routing requests from a single dashboard to various external AI backends using configurable base URLs and API keys. The platform is delivered as a cross-platform progressive web application, allowing for installation on Linux, Windows, and MacOS. It distinguishes itself by consolidating text, image, audio, and video generative controls into a standardized interface, su

ai-toolkit is a diffusion model training toolkit designed for fine-tuning image and video generation models. It functions as a containerized model trainer and GPU training job manager, providing the infrastructure to orchestrate dependencies and manage training processes on remote GPU hardware. The system utilizes low-rank adaptation techniques, including LoRA and LoKr weight optimization, to reduce the hardware requirements for model training. It distinguishes itself through a web-based training controller that allows for the monitoring and modification of hyperparameters, secured by token-b

This project is a LLaMA fine-tuning framework and training pipeline designed for instruction tuning using low-rank adaptation. It provides a system for adapting large language models through a portable, containerized machine learning environment and a web-based inference interface. The framework enables the training of low-rank adapters and the subsequent merging of these weights back into base models to create standalone checkpoints. It includes utilities for defining and formatting prompt templates to ensure consistent data structures during the fine-tuning and inference processes. The pro

This project is a language model finetuning framework designed to adapt large language models to specific datasets using supervised fine-tuning and low-rank adaptation. It serves as a distributed training manager that coordinates workloads and synchronizes gradients across multiple processing units to scale performance. The framework includes a specialized toolkit for low-rank adaptation to update a subset of model weights, reducing memory and hardware requirements. It provides capabilities for instruction fine-tuning, domain adaptation, and the optimization of function calling to improve how

This project is a toolkit for fine-tuning and managing text-to-image diffusion models. It focuses on low-rank adaptation to create small, portable weight files that customize model styles and behaviors without modifying the entire base model. The project provides specialized utilities for model distillation using singular value decomposition to extract adapters from fully trained models, as well as tools for blending and merging multiple adapters through weight interpolation. It includes capabilities for subject inversion and pivotal tuning to increase the visual fidelity of specific identiti

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset