OpenManus/OpenManus-RL

Features

• Distributed Language Agent RL Workflows - Provides a distributed RL training pipeline for language agents using parameter sharding and asynchronous trajectory generation.
• Branching Reasoning Explorations - Systematically explores branching decision paths using Tree-of-Thoughts and Monte Carlo Tree Search.
• Reward Modeling - Includes a platform for developing specialized reward models that quantify performance signals from agent-environment interactions.
• Trainer Coordination - Coordinates environment initialization, worker group scaling, and policy updates across multiple compute nodes.
• Distributed Rollout Systems - Ships a coordinated rollout system that collects interaction trajectories across multiple worker nodes.
• Distributed Training Sharding - Shards model parameters and optimizer states across multiple compute nodes to enable large-scale training beyond single-GPU memory.
• Reinforcement Learning Alignment - Trains language models as agents using reinforcement learning objectives such as PPO for task performance.
• Parallel Trajectory Generation - Produces sequences of agent-environment interaction trajectories across distributed nodes using high-performance inference engines.
• Advantage Estimation - Implements Generalized Advantage Estimation to reduce variance during policy updates by calculating relative action values.
• Reasoning Optimization - Implements advanced reasoning optimization using strategies like Tree-of-Thoughts and Monte Carlo Tree Search to improve model decision-making.
• Branching Trajectories - Implements advanced reasoning strategies such as Tree-of-Thoughts and Monte Carlo Tree Search to explore branching decision paths.
• Reinforcement Learning Optimizers - Implements algorithms for optimizing model policies based on reward signals to align agent behavior with goals.
• PPO Implementations - Implements Proximal Policy Optimization using clipped surrogate objectives to stabilize language model weight updates.
• Interaction Trajectory Generation - Generates interaction datasets and reasoning paths from environments for model reinforcement learning.
• Remote Environment Orchestration - Provides a system for scaling and managing parallel containerized instances to serve as training environments for AI agents.
• Agent Action Space Exploration - Uses search strategies like Monte Carlo Tree Search to navigate large operational environments and optimize action selection.
• Custom Agent and Environment Definitions - Connects language models to specific task environments using custom agent and environment definitions.
• Reasoning Path Scaling - Allows adjusting the complexity and number of reasoning paths during test-time inference to solve harder tasks.
• Reasoning Strategies - Implements reasoning strategies like Graph-of-Thoughts to improve the efficiency and robustness of planning trajectories.
• Reinforcement Learning Reward Systems - Combines multiple reward sources into a single utility signal to guide the agent toward specific goals.
• Reward Shaping - Computes cumulative reward scores and applies shaping techniques to stabilize the RL training process.
• Training Data Curators - Curates high-quality reasoning datasets by organizing interaction trajectories across multiple domains to reduce hallucinations.
• Agent Task Environments - Links agent classes to isolated conda specifications and automated setup scripts for task-specific environments.
• Conda Environment Registries - Manages the registration of custom conda-based task environments to orchestrate parallel agent rollouts.
• Reasoning Models - Reinforcement learning for agentic reasoning.

Star history