huggingface/alignment-handbook
Alignment Handbook
Ce projet est un framework d'alignement et une suite de pipelines pour l'entraînement de modèles de langage via le fine-tuning supervisé et l'optimisation par préférence. Il fournit des outils pour exécuter un entraînement distribué à grande échelle sur plusieurs GPU et nœuds de calcul, ainsi qu'un système pour mesurer l'utilité du modèle et la qualité des dialogues via des benchmarks mono-tour et multi-tours.
Le framework inclut des outils spécialisés pour l'optimisation directe par préférence (DPO) afin d'affiner le comportement du modèle en utilisant des données appariées sans nécessiter de modèle de récompense séparé. Il prend également en charge l'alignement par IA constitutionnelle et l'entraînement de modèles de récompense pour classer et noter les réponses selon des critères de préférence.
Le projet couvre des capacités plus larges pour le mélange de jeux de données, le fine-tuning efficace en paramètres via l'adaptation de bas rang (LoRA), et l'optimisation par échantillonnage par rejet. Il gère le cycle de vie de l'entraînement via des recettes basées sur la configuration et fournit des systèmes pour diffuser des métriques de performance en temps réel vers des tableaux de bord externes.
Features
- Preference Alignment - Trains language models to follow human or AI preferences through supervised fine-tuning and preference optimization.
- Reward Modeling - Provides capabilities for training reward models to distinguish and score different model responses.
- Preference Alignment - Refines model behavior using human and AI feedback to ensure outputs match specific preference criteria.
- Direct Preference Optimization - Refines model behavior using paired preference data to align outputs without a separate reward model.
- Distributed Training Configurations - Includes configurations for scaling model training workloads across multiple GPUs and compute nodes.
- Large Scale Training Suites - Provides comprehensive frameworks for orchestrating distributed multi-node training of large language models.
- Distributed Training - Scales large-scale model training across multiple GPUs and compute nodes using distributed orchestration.
- Alignment Frameworks - Provides a full suite of tools and recipes for training and aligning language models using SFT and preference optimization.
- Multi-Stage Pipelines - Sequences supervised fine-tuning and preference optimization to iteratively refine model behavior.
- Supervised Fine-Tuning - Provides a process for training language models on curated instruction-response pairs for task adaptation.
- Supervised Fine-Tuning Frameworks - Provides a comprehensive framework for aligning pretrained models to specific tasks and behaviors using labeled datasets.
- Training Recipes - Uses standardized training recipes to define and reproduce exact hyperparameters and model training parameters.
- Chat Model Evaluations - Includes a system for assessing instruction-tuned chat models on conversational ability and human alignment.
- Distributed Computing - Coordinates large-scale model alignment tasks across multiple GPUs and compute nodes.
- Configuration-Driven Pipelines - Provides reproducible training pipelines defined via structured recipe configuration files.
- Constitutional Alignments - Supports alignment based on structured constitutional principles and recipes.
- LLM Benchmarking - Evaluates alignment quality and model helpfulness using standardized benchmarks and task configurations.
- Training Optimization Strategies - Implements acceleration strategies and distributed processing to optimize training across varying hardware setups.
- Low-Rank Adaptation - Implements low-rank adaptation to reduce memory requirements during large-scale language model optimization.
- Rejection Sampling Optimizations - Implements rejection sampling to improve the quality of fine-tuned models by selecting the best responses.
- Parameter Efficient Fine-Tuning - Implements low-rank adaptation (LoRA) to reduce memory requirements during large-scale model optimization.
- Algorithm Comparisons - Provides a framework for evaluating the performance of different alignment methods across various models.
- Dataset Blending - Combines multiple datasets with weighted sampling and formats them into chat templates for training.
- Training Metric Streaming - Streams real-time training and evaluation metrics to external monitoring dashboards.
- Multi-Turn Dialogue Benchmarking - Measures model helpfulness and quality using single-turn and multi-turn conversational benchmarks.
- Reinforcement Learning - Directly distills alignment preferences into language models.
