facebookresearch/Horizon
Horizon
Horizon is a reinforcement learning platform designed for training, evaluating, and deploying agents and contextual bandits using historical data. It serves as an off-policy engine and offline policy evaluation tool, allowing decision-making policies to be optimized and tested without the need for a live simulator.
The framework specializes in recommendation system optimization, specifically using slating-based reinforcement learning to optimize the ordering and sequencing of multiple recommendations. It also functions as a contextual bandit framework that manages the balance between exploration and exploitation through uncertainty-based sampling.
The platform's capabilities cover offline reinforcement learning, counterfactual policy evaluation via importance sampling, and behavioral policy bootstrapping. It supports the training of agents for discrete and parametric action spaces, utilizing distributional value estimation and actor-critic algorithms.
Features
- Offline Reinforcement Learning - Implements a framework for training and optimizing reinforcement learning agents using pre-collected historical datasets.
- Off-Policy Frameworks - Provides a complete off-policy reinforcement learning engine for training agents using historical transition data.
- Actor-Critic Architectures - Implements actor-critic architectures to optimize decision-making policies through a separate value-estimating critic and action-selecting actor.
- Contextual Bandit Frameworks - Provides a framework for managing contextual bandits, balancing exploration and exploitation using uncertainty-based strategies.
- Off-Policy Data Engines - Calculates value functions and gradients using historical transition datasets to decouple agent learning from live interaction.
- Slate Recommenders - Optimizes the ordering and sequencing of multiple recommendations using specialized slating-based RL.
- Reinforcement Learning - Functions as a comprehensive platform for training, evaluating, and deploying reinforcement learning agents and contextual bandits.
- Reinforcement Learning Training - Provides capabilities for training decision-making agents using off-policy and actor-critic algorithms.
- RL Training Workflows - Optimizes decision-making agents for discrete and parametric action spaces using off-policy and actor-critic algorithms.
- Off-Policy Engines - Ships an off-policy engine that optimizes policies using actor-critic and distributional algorithms without a live simulator.
- Offline Policy Evaluators - Provides methods for estimating the performance of target policies using logged trajectory datasets without requiring a live simulator.
- Agent Action Spaces - Supports decision-making models that handle both fixed discrete choices and continuous-valued parametric action spaces.
- Bootstrapping Techniques - Initializes learning agents by mimicking action distributions from stable logging policies to ensure safe training starts.
- Policy Bootstrapping - Clones behavior from existing logging policies to create a stable starting point for training new learning agents.
- Uncertainty-Based Sampling - Balances exploration and exploitation in multi-arm problems by selecting actions based on reward confidence intervals.
- Distributional Reward Modeling - Predicts the full probability distribution of future rewards to better handle uncertainty in decision-making.
- Distributional Value Estimation - Predicts the full probability distribution of rewards rather than a single scalar mean for more robust learning.
- Structured Action Spaces - Treats sequences of recommendations as a single combined action within the model to optimize ordering.
- Reinforcement Learning - Facebook RL framework.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
