SkyworkAI/SkyReels-V2
SkyReels V2
SkyReels-V2 is a video generation system that creates, extends, and refines video clips from text descriptions, images, or both. It operates as a diffusion-based video generation model that can produce videos of any duration by denoising frames sequentially, with each new frame conditioned on the ones that came before it. The system supports generating videos from scratch using text prompts, starting from a single image and producing subsequent frames, or constraining both the first and last frames to match user-provided images.
What distinguishes SkyReels-V2 is its combination of infinite-length video generation, frame-level control, and motion quality refinement through reinforcement learning. The system can extend videos indefinitely by denoising tokens at independent noise levels per frame, enabling seamless continuation of footage beyond typical length limits. It also applies direct preference optimization on preference pairs to train the model toward physically plausible, large-motion sequences, improving temporal coherence and motion quality. A prompt expansion language model automatically expands brief text descriptions into more detailed prompts, while a vision-language captioning model generates detailed textual descriptions of video content including shot types and camera movements.
The system includes multi-GPU pipeline parallelism that distributes frame batches across multiple GPUs to reduce end-to-end inference time for large-scale outputs. It also supports video extension, appending new frames to an existing clip by conditioning on its last frames for seamless continuation.
Features
- Video Clip Generators - Creates video clips from text descriptions, images, or both, using a diffusion-based denoising process.
- Video Motion Refiners - Improves motion quality in video generation by applying direct preference optimization on preference pairs.
- Preference-Based Motion Refiners - Refines the model's ability to generate physically plausible, large-motion sequences by training on preference pairs and applying direct preference optimization.
- Text-to-Video Generators - Synthesizes dynamic video content from descriptive text prompts using a diffusion-based denoising process.
- Image-Conditioned Video Generators - Generates video frames conditioned on a starting image, using the image as the first frame and producing subsequent frames from a text prompt.
- Per-Frame Noise Level Schedulers - Denoises tokens at different noise levels independently per frame, enabling infinite-length video generation by extending sequences frame-by-frame.
- Motion Quality Refiners - Applies direct preference optimization on preference pairs to train the model toward physically plausible, large-motion sequences.
- Image-to-Video Generation - Produces a video starting from a single input image, using the image as the first frame and generating subsequent frames based on a text prompt.
- Infinite-Length Generators - Creates videos of any duration by extending a sequence frame-by-frame with a model that denoises tokens at independent noise levels.
- Video Continuation Tools - A model that appends new frames to an existing video clip by conditioning on its last frames, enabling seamless continuation of footage.
- Autoregressive Frame Denoisers - Generates video by denoising each frame sequentially, conditioning each new frame on the previous ones to produce coherent temporal sequences.
- Frame-Controlled Generators - A tool that produces videos with user-specified start and end frames, ensuring the output begins and ends with specific visuals.
- Prompt Expansion - Expands brief text descriptions into detailed prompts using a language model for better video alignment.
- Multi-GPU Video Inference Accelerators - Distributes video generation workloads across multiple GPUs to reduce inference time for large-scale outputs.
- Vision-Language Models - Uses a vision-language model to automatically generate detailed textual descriptions of video content, including shot types and camera movements.
- Video Captioning - Generates detailed text descriptions of video content, including shot types, camera movements, and subject actions, using a vision-language model.
- Frame Constraint Encoders - Encodes both the first and last user-provided images as constraints, ensuring the generated video begins and ends with specific visuals.
- Frame-Batch Distributors - Splits the video generation workload across multiple GPUs by distributing frame batches, reducing end-to-end inference time through parallel processing.
- Foundation Models - Foundation model for infinite-length film generation.
- Video Generation - High-fidelity video generation and storytelling model.
- Video Generation Models - High-performance video generation model.
