Awesome GitHub RepositoriesInference Acceleration

InstantID is a diffusion-based identity preservation framework designed for zero-shot image generation. It allows for the synthesis of images featuring a specific person's facial identity using a single reference photo without requiring additional model training or fine-tuning. The project distinguishes itself through the use of consistency model distillation to accelerate inference, reducing the number of steps needed to produce high-quality results. It combines identity-preserving feature extraction with multi-modal prompt integration to merge visual embeddings from a reference image with t

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

DiffSinger es un sintetizador vocal de IA y generador de audio neuronal diseñado para producir canto y habla de alta fidelidad. Funciona como un sistema de texto a voz y una herramienta de síntesis de voz cantada basada en difusión que transforma texto y tono en audio audible. El sistema utiliza un mecanismo de difusión superficial y refinamiento iterativo de ruido para generar interpretaciones vocales realistas. Incorpora plugins de muestreo especializados y solucionadores numéricos para acelerar la inferencia y reducir el tiempo requerido para generar voces sintéticas. El proyecto cubre el modelado acústico, la síntesis de mel-espectrogramas y la reconstrucción de vocoder neuronal para convertir texto en formas de onda de audio en el dominio del tiempo. También incluye capacidades para la mejora vocal sintética para mejorar la calidad sónica de las grabaciones.

HunyuanVideo-1.5 is a video generation foundation model and text-to-video diffusion framework. It utilizes a latent video diffusion model and a spatio-temporal transformer architecture to generate high-definition video sequences from text descriptions and images. The project enables cinematic camera control for directing pans and tilts and provides image-to-video animation capabilities. It supports visual style adaptation through low-rank adaptation tuning and uses a language model for prompt refinement to improve visual alignment. The model covers high-resolution video upscaling via a super

HunyuanDiT es un modelo generativo bilingüe de texto a imagen y un generador de imágenes de transformador de difusión. Utiliza un sistema de difusión latente para sintetizar imágenes de alta resolución a partir de prompts de texto, con un enfoque específico en la comprensión y generación de contenido a partir de descripciones en chino e inglés. El proyecto presenta una arquitectura de transformador de resolución múltiple y un espacio de incrustación bilingüe para mapear diferentes scripts en un área semántica compartida. Admite el refinamiento iterativo de imágenes en múltiples turnos, lo que traduce el diálogo conversacional en prompts actualizados para modificar progresivamente el contenido visual. El sistema incluye capacidades para el subtitulado automático de imágenes, restricciones estructurales de imagen para el control del diseño y ajuste fino de los pesos del modelo para adaptar el generador a conjuntos de datos o estilos artísticos específicos. Las optimizaciones de rendimiento incluyen la destilación del modelo para la aceleración de la inferencia y el soporte para la ejecución en hardware con poca memoria de video.

FastVideo is a comprehensive system for accelerated video generation, serving as a video generation inference engine, a video diffusion training framework, and a modular pipeline orchestrator. It provides a distributed transformer optimizer and a distillation toolkit designed to reduce denoising steps and model complexity to increase frame rates. The project distinguishes itself through specialized acceleration techniques, including joint distillation and sparse attention training. It implements low-step video generation and weight quantization to FP8 or FP4 precision to increase throughput a

MAGI-1 is an autoregressive video generation model designed to synthesize high-resolution video sequences from text prompts and image references. It functions as a generative system for text-to-video, image-to-video, and video-to-video transformations. The model utilizes an autoregressive architecture that treats spatio-temporal patches as a sequence of discrete tokens to maintain temporal motion. It employs a variational autoencoder to compress the spatial and temporal dimensions of video data and uses distillation-based step scaling to allow for inference budget control. The system integra

TurboDiffusion is a video diffusion inference engine and generator designed to create high-resolution videos from text prompts and images. It provides a runtime environment for executing optimized diffusion model checkpoints with a focus on reducing latency and GPU memory usage. The project features a specialized training framework for aligning sparse-linear attention models with pretrained full-attention models. This system includes capabilities for sparse attention parameter merging and sparse-linear model alignment to reduce computational costs during inference while maintaining output qua