fudan-generative-vision/champ
Champ
Champ is a generative vision system and controllable image-to-video generator designed for human image animation. It uses a diffusion-based video synthesizer and 3D parametric guidance to transform a single reference image into a consistent sequence of motion based on external driving data.
The framework distinguishes itself through a human pose transfer system that employs 3D body parametric extraction and coordinate-space alignment. This allows the model to map motion from a driving video to a reference person by adjusting for body scales and camera perspectives using depth and semantic condition maps.
The system covers a broad range of capabilities including motion data refinement, parametric motion transfer, and a two-stage vision model training pipeline. These processes ensure structural consistency and temporal stability across generated human animations.
Features
- Video Diffusion Models - Uses a video diffusion model to generate temporally consistent human animations.
- 3D Parametric Human Animation Models - Provides a generative vision system using 3D parametric guidance to animate human figures.
- Human Pose Transfer Frameworks - Extracts motion from driving videos and applies it to reference images using depth and semantic maps.
- Condition Map Conditioning - Steers the diffusion process using a combination of rendered depth, pose, and semantic maps.
- Motion Transfer Models - Maps motion from a driving video to a target person by aligning body scales and perspectives.
- Parametric Human Model Fitting - Fits 3D body models to reference images and driving videos to create structured motion data.
- Structural Guidance - Implements structural guidance by using 3D human body parameters to enforce spatial consistency during animation.
- Image-to-Video Animators - Transforms a single reference image into a moving video based on an external motion source.
- Pose Guidance - Controls human movement and pose in generated videos using 3D parametric body models.
- Condition Map Rendering - Renders processed 3D body data into visual condition maps to guide the animation process.
- Vision Model Training - Optimizes generative animation models using datasets of depth, pose, and semantic maps.
- Visual Identity Consistency - Ensures the generated human animations maintain visual identity and structural stability across frames.
- Parametric Motion Refinement - Smooths predicted body parametrics and aligns driving motion to match reference image figure and camera space.
- Shared Coordinate Space Alignments - Aligns driving pose data into the specific 3D coordinate frame of the reference image.
- Generative Stability Pipelines - Employs a two-stage training process optimizing on static images first and then video sequences for stability.
- Controllable Generation - Animates human images using 3D parametric guidance.
