Trlx

trlx est une bibliothèque d'apprentissage par renforcement et un framework d'entraînement conçu pour aligner les grands modèles de langage en utilisant le feedback humain. Il sert d'entraîneur distribué et d'orchestrateur de calcul pour scaler des modèles à haut nombre de paramètres sur plusieurs GPU et nœuds.

Le projet fournit des outils pour l'apprentissage par renforcement à partir de feedback humain et l'alignement de modèles. Il implémente l'optimisation basée sur un modèle de récompense et l'optimisation de politique proximale (PPO) pour affiner le comportement du modèle en fonction de récompenses orientées vers des objectifs ou de jeux de données étiquetés par des humains.

Le framework couvre des stratégies d'entraînement distribué, notamment le parallélisme de modèle, le sharding de paramètres et la synchronisation de gradients multi-nœuds. Il intègre également des contraintes comme la divergence KL pour gérer la dérive du modèle pendant le processus d'apprentissage par renforcement.

Features

RLHF Alignment Algorithms - Implements RLHF alignment algorithms like PPO to align model outputs with human preferences.
Reward Modeling - Provides reward modeling capabilities to evaluate and score text generation for preference alignment.
Reinforcement Learning Fine-Tuning - Fine-tunes language models using reward signals from human feedback and RL alignment algorithms.
Language Model Trainers - Serves as a specialized trainer for updating the parameters of large language models across distributed nodes.
Distributed Training - Scales the training of high-parameter language models across multiple compute nodes to manage memory and time.
Reinforcement Learning Optimizers - Optimizes language model behavior and performance using reward functions and RL algorithms.
Transformer Reinforcement Learning Libraries - Provides a library of tools for fine-tuning and aligning transformer models using reinforcement learning.
RLHF Training Pipelines - Offers end-to-end training pipelines that integrate reward modeling with PPO-based policy optimization.
Proximal Policy Optimization Alignment - Implements Proximal Policy Optimization with a clipped objective function to ensure stable reinforcement learning updates.
Distributed Model Parallelism - Employs distributed model parallelism to partition computational workloads across multiple GPUs for massive models.
Distributed Gradient Synchronization - Coordinates gradient updates across a cluster of machines using collective communication patterns to maintain consistency.
Data Parallelism - Implements data parallelism to split training batches across multiple GPU nodes for faster model convergence.
Distributed Device Orchestration - Orchestrates computational loads and memory across multiple hardware devices during large-scale model refinement.
Distributed Parameter Sharding - Partitions large model tensors across multiple compute nodes to fit high-parameter architectures in memory.
KL-Divergence Penalties - Uses KL-divergence penalties to prevent the RL-tuned policy from drifting too far from the reference model.
Open Source Models - Facilitates distributed training with human feedback.
Reinforcement Learning Tools - Framework for fine-tuning models with reinforcement learning.
RLHF Frameworks - Distributed training framework for fine-tuning models with PPO and ILQL.

Historique des stars

carperaitrlx

Name: carperai/trlx
Author: carperai

View on GitHub

4,749 stars484 forksPythonMIT12 vues

Trlx

Features

RLHF Alignment Algorithms - Implements RLHF alignment algorithms like PPO to align model outputs with human preferences.
Reward Modeling - Provides reward modeling capabilities to evaluate and score text generation for preference alignment.
Reinforcement Learning Fine-Tuning - Fine-tunes language models using reward signals from human feedback and RL alignment algorithms.
Language Model Trainers - Serves as a specialized trainer for updating the parameters of large language models across distributed nodes.
Distributed Training - Scales the training of high-parameter language models across multiple compute nodes to manage memory and time.
Reinforcement Learning Optimizers - Optimizes language model behavior and performance using reward functions and RL algorithms.
Transformer Reinforcement Learning Libraries - Provides a library of tools for fine-tuning and aligning transformer models using reinforcement learning.
RLHF Training Pipelines - Offers end-to-end training pipelines that integrate reward modeling with PPO-based policy optimization.
Proximal Policy Optimization Alignment - Implements Proximal Policy Optimization with a clipped objective function to ensure stable reinforcement learning updates.
Distributed Model Parallelism - Employs distributed model parallelism to partition computational workloads across multiple GPUs for massive models.
Distributed Gradient Synchronization - Coordinates gradient updates across a cluster of machines using collective communication patterns to maintain consistency.
Data Parallelism - Implements data parallelism to split training batches across multiple GPU nodes for faster model convergence.
Distributed Device Orchestration - Orchestrates computational loads and memory across multiple hardware devices during large-scale model refinement.
Distributed Parameter Sharding - Partitions large model tensors across multiple compute nodes to fit high-parameter architectures in memory.
KL-Divergence Penalties - Uses KL-divergence penalties to prevent the RL-tuned policy from drifting too far from the reference model.
Open Source Models - Facilitates distributed training with human feedback.
Reinforcement Learning Tools - Framework for fine-tuning models with reinforcement learning.
RLHF Frameworks - Distributed training framework for fine-tuning models with PPO and ILQL.

Alternatives open source à Trlx

Projets open source similaires, classés selon le nombre de fonctionnalités partagées avec Trlx.

lvwerra/trl
lvwerra/trl
18,718Voir sur GitHub
This project is a transformer post-training toolkit and reinforcement learning library designed to align language model behavior with human preferences. It provides a framework for managing the transition from supervised fine-tuning to reinforcement learning and preference optimization. The library distinguishes itself through a specialized focus on preference optimization and reward modeling, enabling the adjustment of model outputs based on preferred versus rejected examples. It also includes capabilities for training agents within controlled sandbox environments using task suites and verif
Python
Voir sur GitHub18,718
infrasys-ai/aiinfra
Infrasys-AI/AIInfra
7,414Voir sur GitHub
Jupyter Notebookaiinfraaisystem
Voir sur GitHub7,414
internlm/xtuner
InternLM/xtuner
5,150Voir sur GitHub
xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised fine-tuning, and the optimization of vision-language multimodal models. It serves as a distributed training accelerator and a specialized framework for scaling Mixture-of-Experts models and aligning model behavior through reinforcement learning from human feedback. The project distinguishes itself through advanced memory and compute optimizations, such as sequence parallelism for ultra-long context windows and interleaved pipeline parallelism to reduce GPU idle time. It provide
Pythonagentdeepseek-v3gpt-oss
Voir sur GitHub5,150
inclusionai/areal
inclusionAI/AReaL
3,559Voir sur GitHub
AReaL is a system for agent orchestration, distributed model training, and parameter-efficient tuning. It provides a framework for developing multi-turn reasoning agents and training large models using reinforcement learning from human feedback. The project implements a toolkit for improving the visual reasoning and geometry problem solving capabilities of vision-language models. It utilizes a memory-efficient tuning system to optimize mathematical and reasoning models across different inference backends. The infrastructure supports large-scale training through tensor, pipeline, and expert p
Pythonagentllmllm-agent
Voir sur GitHub3,559

Voir les 30 alternatives à Trlx

Questions fréquentes

Que fait carperai/trlx ?

Quelles sont les fonctionnalités principales de carperai/trlx ?

Les fonctionnalités principales de carperai/trlx sont : RLHF Alignment Algorithms, Reward Modeling, Reinforcement Learning Fine-Tuning, Language Model Trainers, Distributed Training, Reinforcement Learning Optimizers, Transformer Reinforcement Learning Libraries, RLHF Training Pipelines.

Quelles sont les alternatives open-source à carperai/trlx ?

Les alternatives open-source à carperai/trlx incluent : lvwerra/trl — This project is a transformer post-training toolkit and reinforcement learning library designed to align language… infrasys-ai/aiinfra. internlm/xtuner — xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised… inclusionai/areal — AReaL is a system for agent orchestration, distributed model training, and parameter-efficient tuning. It provides a… hiyouga/easyr1 — EasyR1 is a distributed model training system and reinforcement learning framework for large language and… openrlhf/openrlhf — OpenRLHF is a training framework and alignment library designed for reinforcement learning from human feedback across…

Trlx

Features

Historique des stars

Trlx

Features

Alternatives open source à Trlx

lvwerra/trl

Infrasys-AI/AIInfra

InternLM/xtuner

inclusionAI/AReaL

Questions fréquentes

Historique des stars

Questions fréquentes

Alternatives open source à Trlx

lvwerra/trl

Infrasys-AI/AIInfra

InternLM/xtuner

inclusionAI/AReaL