Reinforcement Learning Agent Libraries

Isaac Lab is an open-source framework for training robot policies in physically simulated environments, supporting both single-agent and multi-agent reinforcement learning. It is built on an Omniverse-PhysX simulation backend that models rigid bodies, articulated systems, deformable objects, and sensors, and provides a task-based environment configuration system where each training environment is defined as a modular class specifying observation spaces, action spaces, reward functions, and termination conditions. The framework distinguishes itself through an RL-library abstraction layer that wraps multiple reinforcement learning libraries behind a unified training interface, enabling drop-in swaps between RL-Games, RSL-RL, SKRL, and Stable-Baselines3. It includes a policy distillation pipeline for compressing large teacher policies into smaller student networks, a multi-agent training orchestrator for cooperative or competitive algorithms, and a callback weak-reference pattern that prevents memory leaks by allowing Python objects to be garbage collected when no longer referenced. The system also manages GPU pipeline buffers dynamically to prevent overflow errors and provides a TensorBoard metric logging system for structured training data visualization. Isaac Lab offers over 30 pre-built training environments for tasks including locomotion, manipulation, assembly, motion imitation, and multirotor control, with support for domain randomization and a library of more than 16 robot models including manipulators, quadrupeds, and humanoids. The framework includes tools for simulation debugging and optimization, such as crash log root cause analysis, GPU pipeline buffer overflow resolution, and physics simulation stabilization using the PhysX Visual Debugger. It also supports cloud training deployment, agent performance video recording, and trained policy playback for evaluation.

IsaacGymEnvs is a GPU-accelerated physics sandbox and robotics policy training suite designed for reinforcement learning. It serves as a vectorized robotic simulator that runs thousands of parallel environments on GPUs to accelerate the training of neural networks. The project provides a sim-to-real transfer framework that utilizes domain randomization and physics variations to ensure policies trained in simulation are robust enough for deployment on real hardware. It distinguishes itself through a high-performance architecture that uses tensor-based state management to handle observations and rewards as contiguous GPU memory buffers. The suite covers diverse robotic domains, including locomotion for quadruped and humanoid agents, dexterous manipulation for robotic hands, aerial navigation for quadcopters, and robotic arm control. It further supports specialized tasks such as human motion imitation and robotic assembly simulation using signed distance field collision detection. The system includes infrastructure for distributed GPU training, population-based hyperparameter optimization, and a framework for defining custom reinforcement learning tasks via base-class inheritance.

Stable-baselines3 is a reinforcement learning library built on the PyTorch deep learning framework. It provides a collection of reliable, standardized implementations of reinforcement learning algorithms designed for training, testing, and benchmarking agent policies in diverse simulated environments. The library functions as an agent training toolkit that emphasizes modularity and reproducibility. It features a unified environment interface and supports vectorized execution to accelerate data collection across multiple simulation instances. Users can customize neural network architectures, feature extractors, and policy definitions to suit specific observation and action spaces, while built-in tools for deterministic seeding ensure consistent results across training runs. Beyond core training, the project includes comprehensive utilities for managing the agent lifecycle. This encompasses memory-efficient experience replay buffering, advanced exploration strategies for continuous control, and automated monitoring of performance metrics. The framework also supports the export and distribution of trained models, facilitating collaboration and deployment across various hardware and runtime environments.

Gym is a reinforcement learning environment toolkit and agent simulation framework. It provides a standardized API and a universal communication interface that defines how learning agents interact with simulation environments through actions and observations. The project includes a benchmark environment suite and a diverse library of pre-configured simulation worlds, including physics engines and classic control tasks. It enables the creation of custom simulation environments to train agents in specific operational scenarios while ensuring reproducibility across different learning algorithms. The framework manages state-transition simulations, mapping raw observation data and translating agent decisions into compatible action values. It utilizes environment wrappers to modify observations or rewards and a versioned registry to maintain consistency across benchmarks.

rllm is an asynchronous reinforcement learning framework for training language agents. It provides a unified pipeline that runs the same agent code for both evaluation and training, automatically capturing traces for gradient computation. The framework supports distributed reinforcement learning across multiple GPUs and nodes using pluggable backends, and executes agents in isolated sandboxes—either locally or in the cloud—for safe and scalable rollout collection. It trains agents built with LangGraph, SmolAgents, OpenAI Agents SDK, or custom frameworks without requiring core logic changes. The framework distinguishes itself through native multi-agent training orchestration, where collaborative workflows such as solver-judge pairs learn from shared or competing trajectories with differentiated rewards per agent role. It includes a library of over 50 curated benchmarks spanning math, code, QA, and vision, and provides a suite of pre-built reward functions and graders. Performance optimizations include pre-provisioned sandbox queues and startup snapshot caching to reduce rollout latency, and a transparent HTTP proxy captures token-level data from any inference request without modifying agent code. Beyond its core training capability, rllm offers a CLI for launching training and evaluation jobs with automated dataset handling, and supports progressive context length scaling, parameter-efficient fine-tuning via LoRA, and multimodal model training. It integrates AI-backed run analysis, real-time web dashboard monitoring, and full-text search across training artifacts. The framework’s pluggable backend interface and environment-variable-driven configuration allow switching between Ray-distributed, managed-service, or single-machine backends without code changes, and its curated dataset management and custom dataset integration methods make it straightforward to bring new tasks into the training workflow.

verl is a distributed training system designed for large language model alignment and reinforcement learning. It provides a framework for executing post-training pipelines, including supervised fine-tuning and reinforcement learning from human feedback, to refine model behavior and agentic capabilities. The system utilizes a hybrid training and inference engine that optimizes memory and communication when switching between model generation and gradient updates. It supports multi-modal reinforcement learning for models processing both image and text data, and implements algorithms such as PPO and GRPO to align models using reward signals. The architecture focuses on distributed scaling through expert parallelism, device-aware placement mapping, and memory resharding. It further reduces resource overhead via low-rank adaptation and decoupled computation dataflows, while providing modular interfaces to integrate with various training and inference engines. The project includes tools for experiment tracking to log training metrics and performance data to external monitoring platforms.

FinRL is a reinforcement learning framework designed for the development, training, and backtesting of automated trading strategies. It functions as a quantitative finance toolkit that integrates deep learning algorithms with financial market simulations to address complex portfolio management and asset allocation tasks. The platform provides an end-to-end pipeline for transforming raw market data into actionable trading models. The project distinguishes itself through a layered, modular architecture that separates data processing, environment simulation, and agent training. This design allows for the creation of standardized market environments that incorporate real-world frictions, such as transaction costs and portfolio constraints, ensuring that strategies are validated against realistic conditions. By utilizing parallel simulation execution, the framework accelerates the training process across diverse asset classes, including stocks and cryptocurrencies. Beyond training, the system supports the full lifecycle of algorithmic trading, from initial data ingestion and feature engineering to performance benchmarking against established quantitative baselines. It includes tools for calculating standard financial metrics, tuning model hyperparameters, and deploying trained agents to live brokerage interfaces for real-time execution. The framework is designed to be extensible, enabling users to swap components or integrate custom reinforcement learning libraries to suit specific research or operational objectives.

This project is a Python-based educational framework designed to simulate reinforcement learning algorithms and environments. It serves as a platform for reproducing classic textbook examples, allowing users to study agent behavior, policy improvement, and the fundamental mechanics of decision-making in controlled settings. The library provides implementations for core reinforcement learning concepts, including temporal difference learning, Monte Carlo episode sampling, and tabular value function approximation. It enables the analysis of specific algorithmic behaviors, such as identifying and mitigating maximization bias, while supporting the exploration of discrete state-space modeling and probabilistic decision-making strategies. Users can engage with various simulation scenarios, ranging from multi-armed bandit modeling to grid world navigation and game-based tasks like tic-tac-toe. These tools facilitate the study of how agents balance exploration and exploitation to maximize cumulative rewards within structured, discrete environments.

Tinker Cookbook is an open-source framework for fine-tuning large language models, supporting supervised learning, reinforcement learning, and parameter-efficient techniques like LoRA adapters. It provides a complete pipeline for aligning models with human preferences through multi-stage RLHF workflows, from supervised fine-tuning through preference optimization to reinforcement learning. The framework distinguishes itself through recipe-based training orchestration, where fine-tuning workflows are defined as composable recipe files that chain data loading, model configuration, and training loops into repeatable pipelines. It includes an async concurrent sampling engine that maximizes throughput during training rollouts and evaluation, and supports multi-agent reinforcement learning with self-play or competitive environments. The system manages model checkpoints through hub-centric weight management, enabling saving, loading, downloading, and publishing to remote hubs for sharing and deployment. Beyond core training, the framework covers hyperparameter sweeping across learning rates, LoRA ranks, and RL parameters to find optimal configurations. It handles vision-language model fine-tuning, prompt distillation into model weights, and multi-turn conversation training. The system includes tools for building, merging, and exporting LoRA adapters for efficient serving and HuggingFace compatibility, along with evaluation capabilities for measuring model performance on standard benchmarks. The documentation provides guidance on configuring training runs, building custom reinforcement learning environments, and diagnosing training issues through AI assistant skills.

Lab is a customizable 3D platform and research testbed designed for training and testing autonomous agents using reinforcement learning. It serves as a spatial AI training simulator where agents can be evaluated through navigation and puzzle-solving tasks. The environment allows for the definition of complex layouts and task behaviors through external scripting, enabling the generation of specific challenges for AI research. It supports both automated training via standard API bindings and manual agent control to validate simulation dynamics. The system utilizes a grid-based spatial representation and converts 3D data into state vectors for agent decision-making. Execution is handled through discrete time-step updates to ensure deterministic behavior during the learning process.

ConvNetJS is a JavaScript deep learning library and neural network training engine designed for client-side machine learning. It functions as a framework for building, training, and running convolutional neural networks directly within a web browser without the need for a backend server. The library specializes in image recognition and pattern analysis using convolutional and pooling layers. It enables the creation of models for classification and regression tasks, as well as the development of reinforcement learning agents that optimize behavior through trial and error in simulated environments. The system provides capabilities for neural network prototyping, image pattern recognition, and the processing of visual data through a sequence of connected layers and non-linear modules.

DeepSpeedExamples is a collection of reference implementations and scripts for training, fine-tuning, and executing inference on large-scale AI models using DeepSpeed optimization. It provides a distributed model training guide and practical workflows for adapting large language models through memory-efficient techniques. The repository includes specialized implementations for pipeline parallelism to handle models exceeding single GPU memory and a suite of examples for ZeRO memory optimization to reduce per-device overhead. It also features standardized test suites for benchmarking the throughput and latency of models running on DeepSpeed inference engines. The project covers broad capability areas including GPU memory optimization, distributed AI benchmarking, and high-performance model inference. It demonstrates the use of weight compression and distributed optimization to scale neural networks across multiple computing nodes.

Keras is a high-level deep learning API used to design, build, and train neural networks for tasks such as computer vision, natural language processing, and time series forecasting. It provides a framework for defining model architectures and optimizing weights through a structured interface. The project is defined by a backend-agnostic design that allows the same model code to run across different compute engines. This multi-backend execution enables users to swap underlying engines to optimize for specific hardware or performance requirements. The system supports distributed model training to scale workloads from local machines to clusters of accelerators. It includes capabilities for managing deep learning data pipelines with diverse dataset formats and provides a pluggable architecture for integrating custom layers, models, and metrics.

Torchtitan is a reference implementation for distributed deep learning built within the PyTorch ecosystem. It provides a framework for training large neural network models across multiple GPUs and nodes by combining several parallelism techniques, including fully sharded data parallelism (FSDP), tensor parallelism, and pipeline parallelism, making it possible to train models that exceed the memory capacity of a single device. The system distinguishes itself through asynchronous checkpointing, which saves model and optimizer state to persistent storage without pausing the training loop, enabling fault tolerance and iterative experimentation. A unified composable parallelism scheduler allows data, tensor, and pipeline parallelism to be orchestrated from a single configuration, while a real-time monitoring tool logs loss, throughput, memory, and other metrics during training runs. The checkpoint format is designed to be directly loadable into conversion tools for subsequent fine‑tuning. Additional capabilities include memory profile–driven autotuning that recommends optimal parallelism configurations, an elastic training coordinator that manages dynamic membership changes in the worker pool, and pipeline execution scheduling that minimises bubble time. These components collectively support large-scale distributed training with both high efficiency and operational flexibility.

Lightning is a PyTorch training framework and distributed AI training orchestrator designed to decouple core research logic from the engineering boilerplate required for model training. It functions as a deep learning workflow manager that automates the process of pretraining and finetuning models across diverse compute environments. The project distinguishes itself by providing a hardware-agnostic training wrapper, allowing the same model code to execute on CPUs, GPUs, or TPUs without modification. It further manages the scaling of workloads from single devices to multi-node clusters and serves as a cloud GPU infrastructure manager with integrated autoscaling and monitoring. The framework covers a broad range of training capabilities, including distributed data parallelism, automatic mixed precision, and state-based checkpoint automation. It also provides tools for production model export and supports custom training loop primitives for specialized model architectures.

Open-r1 is a framework designed for the large-scale training, distillation, and optimization of language models focused on complex reasoning and programming tasks. It provides a comprehensive suite of tools for managing distributed training jobs across multi-node clusters, enabling the development of high-performance models through reinforcement learning and supervised fine-tuning. The project distinguishes itself by integrating secure, containerized code execution environments directly into the training and evaluation lifecycle. By allowing models to run and verify code snippets against test cases, the framework improves accuracy in mathematical and logical problem-solving. It further supports advanced reasoning capabilities through group relative policy optimization and automated synthetic data pipelines, which curate and filter high-quality reasoning traces for model updates. The system utilizes modular, configuration-driven recipes to streamline complex workflows, including data decontamination, dataset composition, and multi-node orchestration. It includes standardized benchmarking tools to measure performance across reasoning and coding domains, ensuring that training processes remain reproducible and data-centric. The framework is built to handle the full lifecycle of model improvement, from initial synthetic data generation to final performance evaluation on high-performance computing clusters.