Awesome GitHub RepositoriesInference Acceleration

InstantID is a diffusion-based identity preservation framework designed for zero-shot image generation. It allows for the synthesis of images featuring a specific person's facial identity using a single reference photo without requiring additional model training or fine-tuning. The project distinguishes itself through the use of consistency model distillation to accelerate inference, reducing the number of steps needed to produce high-quality results. It combines identity-preserving feature extraction with multi-modal prompt integration to merge visual embeddings from a reference image with t

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

DiffSinger ist ein KI-Gesangssynthesizer und neuronaler Audiogenerator, der darauf ausgelegt ist, hochqualitativen Gesang und Sprache zu produzieren. Er fungiert als Text-to-Speech-System und als diffusionsbasiertes Tool zur Synthese von Gesangsstimmen, das Text und Tonhöhe in hörbares Audio transformiert. Das System nutzt einen flachen Diffusionsmechanismus und iterative Rauschverfeinerung, um realistische Gesangsdarbietungen zu generieren. Es integriert spezialisierte Sampling-Plugins und numerische Löser, um die Inferenz zu beschleunigen und die Zeit zu reduzieren, die zur Generierung synthetischer Stimmen erforderlich ist. Das Projekt deckt akustische Modellierung, Mel-Spektrogramm-Synthese und neuronale Vocoder-Rekonstruktion ab, um Text in Zeitbereichs-Audio-Wellenformen zu konvertieren. Es enthält zudem Funktionen zur synthetischen Stimmverbesserung, um die klangliche Qualität von Aufnahmen zu steigern.

HunyuanVideo-1.5 is a video generation foundation model and text-to-video diffusion framework. It utilizes a latent video diffusion model and a spatio-temporal transformer architecture to generate high-definition video sequences from text descriptions and images. The project enables cinematic camera control for directing pans and tilts and provides image-to-video animation capabilities. It supports visual style adaptation through low-rank adaptation tuning and uses a language model for prompt refinement to improve visual alignment. The model covers high-resolution video upscaling via a super

HunyuanDiT ist ein zweisprachiges Text-zu-Bild-Generativmodell und ein Diffusions-Transformer-Bildgenerator. Es verwendet ein latentes Diffusionssystem, um hochauflösende Bilder aus Text-Prompts zu synthetisieren, mit einem besonderen Fokus auf das Verständnis und die Generierung von Inhalten aus chinesischen und englischen Beschreibungen. Das Projekt zeichnet sich durch eine Multi-Resolution-Transformer-Architektur und einen zweisprachigen Embedding-Raum aus, um verschiedene Schriften in einen gemeinsamen semantischen Bereich abzubilden. Es unterstützt iterative, mehrstufige Bildverfeinerungen, bei denen Konversationsdialoge in aktualisierte Prompts übersetzt werden, um visuelle Inhalte schrittweise zu modifizieren. Das System umfasst Funktionen für automatisierte Bildbeschriftungen, strukturelle Bildvorgaben zur Layout-Kontrolle sowie das Fine-Tuning von Modellgewichten, um den Generator an spezifische Datensätze oder künstlerische Stile anzupassen. Performance-Optimierungen beinhalten Model-Distillation zur Beschleunigung der Inferenz sowie Unterstützung für die Ausführung auf Hardware mit geringem Videospeicher.

FastVideo is a comprehensive system for accelerated video generation, serving as a video generation inference engine, a video diffusion training framework, and a modular pipeline orchestrator. It provides a distributed transformer optimizer and a distillation toolkit designed to reduce denoising steps and model complexity to increase frame rates. The project distinguishes itself through specialized acceleration techniques, including joint distillation and sparse attention training. It implements low-step video generation and weight quantization to FP8 or FP4 precision to increase throughput a

MAGI-1 is an autoregressive video generation model designed to synthesize high-resolution video sequences from text prompts and image references. It functions as a generative system for text-to-video, image-to-video, and video-to-video transformations. The model utilizes an autoregressive architecture that treats spatio-temporal patches as a sequence of discrete tokens to maintain temporal motion. It employs a variational autoencoder to compress the spatial and temporal dimensions of video data and uses distillation-based step scaling to allow for inference budget control. The system integra

TurboDiffusion is a video diffusion inference engine and generator designed to create high-resolution videos from text prompts and images. It provides a runtime environment for executing optimized diffusion model checkpoints with a focus on reducing latency and GPU memory usage. The project features a specialized training framework for aligning sparse-linear attention models with pretrained full-attention models. This system includes capabilities for sparse attention parameter merging and sparse-linear model alignment to reduce computational costs during inference while maintaining output qua