magic-research/magic-animate
Magic Animate
Magic Animate is a diffusion model video generator designed for human image animation. It transforms a static human photo into a temporally consistent video by mapping movements from a reference motion clip, acting as a tool to create realistic animations from a single image.
The system ensures visual stability and minimizes flicker through temporal attention injection and motion-controlled noise scheduling. To accelerate the generation of high-resolution video, it includes a distributed GPU inference engine that splits model workloads across multiple graphics cards.
The project covers a comprehensive animation pipeline, including appearance encoding, denoising processes, and a two-stage training regime. It provides both single-GPU and multi-GPU execution paths and includes a Gradio web interface for uploading assets and previewing results.
Features
- Image-to-Video Animators - Transforms static human photos into temporally consistent videos by mapping movements from a reference motion clip.
- Temporal Attention - Injects cross-frame attention layers into the diffusion model to enforce temporal consistency across video frames.
- Cross-Frame Attention Layers - Implements cross-frame attention injection to ensure visual stability and minimize flicker across the generated video sequence.
- Video Diffusion Models - Utilizes a video diffusion model to iteratively denoise latent representations for temporally consistent animation.
- Latent Conditioning Mechanisms - Uses an appearance encoder to provide latent conditioning that steers the denoising process for animation.
- Multimodal Image Encoders - Encodes a static human image into a numerical representation to condition the denoising UNet.
- Sparse-Frame Appearance Encoders - Encodes the person's visual identity from a reference image to maintain consistency across the generated video.
- Temporally Consistent Embeddings - Ensures visual representations remain stable and coherent across consecutive frames to minimize flicker.
- Distributed Model Parallelism - Splits the diffusion model across multiple GPUs by assigning specific subsets of temporal frames to each device.
- Staged Training Pipelines - Implements a staged training strategy that optimizes appearance and temporal modules separately before performing global fine-tuning.
- Multi-GPU Video Inference Accelerators - Distributes video generation workloads across multiple GPUs to reduce inference time for high-resolution output.
- Model Parallelism - Distributes the diffusion workload across multiple GPUs, with each processor handling specific temporal frames.
- Motion-Driven Schedulers - Coordinates the diffusion process by aligning noise patterns with a driving motion sequence to guide frame generation.
- Motion-Aligned - Drives animation by scheduling noise patterns that follow a driving motion sequence to align frames with reference motion.
- Video Generation - Temporally consistent human image animation.
