alibaba/ROLL
ROLL
ROLL is a distributed reinforcement learning framework and model alignment toolkit designed for large language models. It serves as a scalable training pipeline and GPU cluster manager, providing the infrastructure to align model behavior using reinforcement learning algorithms and preference optimization techniques.
The project distinguishes itself through an agentic rollout orchestrator that generates and collects multi-turn interaction trajectories between AI agents and simulated environments. It supports specialized alignment methods including Direct Preference Optimization, reinforcement learning from verifiable rewards, and group-relative reward optimization.
The framework covers a broad range of capabilities for large-scale distributed training, including tensor, pipeline, and expert parallelism to support ultra-large-scale models. It manages hardware resources through GPU multiplexing and disaggregated deployment, while providing tools for automated reward evaluation using code sandboxes and mathematical verification.
Pre-configured environment deployments are provided for different GPU architectures and library versions to accelerate setup.
