Projects sharing features with Facebookresearch DiT

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

This project is a diffusion model framework for training and sampling from denoising probabilistic models to generate images from noise. It functions as a generative image model that creates visual content by iteratively refining random noise into coherent images. The system includes a distributed GPU trainer designed to scale complex neural network architectures across multiple graphics processing units. It also provides an image dataset preprocessor to prepare, scale, and standardize raw image collections for training. The framework covers model training and image generation, utilizing noi

This is a framework for training and sampling diffusion models to generate high-fidelity images, video, and 4D assets. It provides a modular environment for managing generative AI training pipelines, including the handling of datasets, noise sampling, and loss weighting to stabilize the creation of synthetic content. The project features a modular model configuration system that uses YAML-based assembly to define network submodules and conditioners. It also includes a dedicated toolset for AI image watermarking, allowing for the embedding and detection of invisible markers to verify the origi

This project is a comprehensive instructional resource and course for building neural networks using PyTorch. It covers the fundamental building blocks of deep learning, including tensor manipulation, automatic differentiation, and the construction of modular neural network components. The repository serves as a technical guide for several specialized domains. It provides implementation details for computer vision tasks such as image classification, object detection, and semantic segmentation, as well as natural language processing workflows involving transformers, recurrent networks, and gen

HunyuanDiT is a bilingual text-to-image generative model and diffusion transformer image generator. It uses a latent diffusion system to synthesize high-resolution images from text prompts, with a specific focus on understanding and generating content from both Chinese and English language descriptions. The project features a multi-resolution transformer architecture and a bilingual embedding space to map different scripts into a shared semantic area. It supports iterative multi-turn image refinement, which translates conversational dialogue into updated prompts to progressively modify visual

Open-Sora is a video generation framework designed to produce cinematic sequences from text prompts and images. It functions as a generative system that transforms written descriptions or reference images into video content featuring realistic textures and lighting. The project includes a dedicated prompt engineering tool that uses large language models to expand simple user inputs into detailed descriptions. It also features a motion controller for adjusting movement intensity in generated sequences and evaluating motion levels in existing video files. The framework incorporates text-to-vid

Stable-audio-tools is a toolkit for training and deploying latent diffusion models for high-fidelity audio synthesis. It provides a framework for generating audio by iteratively refining noise within a compressed latent space, using specialized encoders to preserve temporal and spectral features of the audio signal. The project features a system for adapting pre-trained audio checkpoints to new datasets through modular initialization and configuration files. It includes utilities for weight extraction and inference model export, which remove training metadata and optimizer states to create li

This is a PyTorch deep learning framework and tool for human motion synthesis that generates 3D character animations from text prompts or action descriptions. It functions as a text-to-motion generator that converts natural language and categorical labels into temporally consistent 3D skeletal movement sequences. The system utilizes a transformer-based diffusion model to iteratively denoise motion data. It includes capabilities for action-conditioned generation, monocular-to-3D motion lifting, and motion sequence editing using text constraints. The framework incorporates geometric motion con

StableCascade is a generative AI system and latent diffusion framework designed for text-to-image synthesis and image-to-image transformations. It utilizes a multi-stage cascade architecture that encodes and decodes images via a latent space to produce high-fidelity visual imagery. The system includes a cascade diffusion pipeline for controlling image structure through inpainting, outpainting, and super-resolution. It also provides a toolkit for image-to-image generation and the creation of image variations using embeddings. The framework supports model optimization through low-rank adaptati

pix2pixHD is a conditional generative adversarial network designed to transform semantic label maps into high-resolution photorealistic images. It functions as a high-resolution image synthesizer and an image-to-image translation model capable of producing synthetic images at 2048x1024 resolution. The system includes a semantic image editor that allows for the modification of high-resolution visuals by updating the underlying semantic label maps. This enables interactive image editing and the generation of photorealistic images based on source images or discrete label maps. The framework pro

This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide

This project is a PyTorch-based generative framework and implementation template for building Generative Adversarial Networks. It provides a collection of foundational toolkits and architectural patterns designed to synthesize high-quality artificial data while focusing on the stability of adversarial neural networks. The framework distinguishes itself through a specialized toolkit for conditional image generation, which integrates discrete labels and auxiliary classification into the training process. It utilizes specific mechanisms to guide the generative process toward target classes by co

This project is a diffusion model training framework and image synthesis pipeline. It provides the tools necessary to train generative models to learn image data distributions through an iterative denoising process. The framework includes a generative model evaluation tool consisting of automated scripts used to measure the quality and accuracy of produced samples. The system covers model training pipelines and performance evaluation for generative diffusion models.

Tiny Universe is an educational monorepo that delivers multiple independent implementations of core AI subsystems as self-contained Jupyter notebooks. It provides from-scratch constructions of foundational architectures including a complete Transformer model built from the original paper specification, a denoising diffusion probabilistic model for image generation, and a ReAct-style autonomous agent framework that equips an LLM with tools for planning and multi-step task execution. The project distinguishes itself by covering the full lifecycle of modern AI systems through hands-on implementa

AudioLCM is a deep learning framework designed for text-to-audio synthesis. It functions as a generative engine that converts written descriptions into high-fidelity audio clips by processing text prompts through latent consistency models. The project distinguishes itself by utilizing latent consistency distillation to enable rapid audio generation. By mapping diffusion trajectories to a single-step consistency function, the system achieves efficient sound synthesis while maintaining the output quality typically associated with iterative diffusion processes. The framework provides a comprehe

This is a classifier-guided diffusion framework for high-fidelity image generation. It implements a cascaded diffusion pipeline that chains a base diffusion model with a dedicated upsampler to progressively increase image resolution in stages, and uses classifier-guided diffusion sampling to steer the reverse diffusion process toward higher-quality outputs. The framework provides tools for training diffusion models from scratch using distributed processes with gradient accumulation, as well as training classifier models that provide gradient-based guidance during sampling. It supports both un

Instruct-pix2pix is an instruction-based image model and PyTorch library designed to modify visual content by following natural language directions. It functions as a diffusion model image editor that applies human-written instructions to existing pictures rather than using traditional text-to-image prompts. The project provides a fine-tunable diffusion framework for adapting pre-trained checkpoints to specific image editing datasets. It includes a synthetic dataset generator that creates paired images and text triplets to train models on various image editing tasks. The system covers a rang

This project is a deep learning framework for AI image super-resolution and facial synthesis. It provides a diffusion model image upscaler and a generative facial image synthesizer capable of transforming low-resolution images into high-resolution outputs using pretrained model weights. The system utilizes iterative diffusion refinement and low-resolution guided sampling to restore fine details and sharpness. It supports both unconditional image generation, where images are created from scratch, and guided resolution enhancement for high-fidelity facial reconstruction. The repository include

This project is a machine learning research automation system designed to manage the full research lifecycle, from idea discovery to final paper submission. It utilizes markdown-based skill templates to execute autonomous research tasks and manage iterative loops of deep review and experimentation. The system distinguishes itself through integrated capabilities for academic communication and integrity auditing. It can automate the generation of LaTeX papers, conference slide decks, and evidence-grounded peer review rebuttals. To ensure rigor, it employs cross-model review routing and adversar

This project is a framework for training and sampling generative models designed to produce high-quality images in few steps. It provides implementations for image generation models that transform random noise into structured visual data through an optimized sampling process. The system specializes in accelerating image generation through consistency distillation and consistency training. It includes tools to transform pre-trained diffusion models into faster versions by distilling knowledge from a teacher model into a student model, as well as methods to train consistency models from scratch

Code for the Paper "Improving Diffusion Model Efficiency Through Patching"

Elucidating The Design Space of Classifier-Guided Diffusion Generation

Official implementation for Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (ICML 2022), and a reimplementation of Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (ICLR 2022)

Speed up Stable Diffusion with this one simple trick!

ICML 2023 official implementation for "Input Perturbation Reduces Exposure Bias in Diffusion Models"