huggingface/alignment-handbook

Features

• Preference Alignment - Trains language models to follow human or AI preferences through supervised fine-tuning and preference optimization.
• Reward Modeling - Provides capabilities for training reward models to distinguish and score different model responses.
• Preference Alignment - Refines model behavior using human and AI feedback to ensure outputs match specific preference criteria.
• Direct Preference Optimization - Refines model behavior using paired preference data to align outputs without a separate reward model.
• Distributed Training Configurations - Includes configurations for scaling model training workloads across multiple GPUs and compute nodes.
• Large Scale Training Suites - Provides comprehensive frameworks for orchestrating distributed multi-node training of large language models.
• Distributed Training - Scales large-scale model training across multiple GPUs and compute nodes using distributed orchestration.
• Alignment Frameworks - Provides a full suite of tools and recipes for training and aligning language models using SFT and preference optimization.
• Multi-Stage Pipelines - Sequences supervised fine-tuning and preference optimization to iteratively refine model behavior.
• Supervised Fine-Tuning - Provides a process for training language models on curated instruction-response pairs for task adaptation.
• Supervised Fine-Tuning Frameworks - Provides a comprehensive framework for aligning pretrained models to specific tasks and behaviors using labeled datasets.
• Training Recipes - Uses standardized training recipes to define and reproduce exact hyperparameters and model training parameters.
• Chat Model Evaluations - Includes a system for assessing instruction-tuned chat models on conversational ability and human alignment.
• Distributed Computing - Coordinates large-scale model alignment tasks across multiple GPUs and compute nodes.
• Configuration-Driven Pipelines - Provides reproducible training pipelines defined via structured recipe configuration files.
• Constitutional Alignments - Supports alignment based on structured constitutional principles and recipes.
• LLM Benchmarking - Evaluates alignment quality and model helpfulness using standardized benchmarks and task configurations.
• Training Optimization Strategies - Implements acceleration strategies and distributed processing to optimize training across varying hardware setups.
• Low-Rank Adaptation - Implements low-rank adaptation to reduce memory requirements during large-scale language model optimization.
• Rejection Sampling Optimizations - Implements rejection sampling to improve the quality of fine-tuned models by selecting the best responses.
• Parameter Efficient Fine-Tuning - Implements low-rank adaptation (LoRA) to reduce memory requirements during large-scale model optimization.
• Algorithm Comparisons - Provides a framework for evaluating the performance of different alignment methods across various models.
• Dataset Blending - Combines multiple datasets with weighted sampling and formats them into chat templates for training.
• Training Metric Streaming - Streams real-time training and evaluation metrics to external monitoring dashboards.
• Multi-Turn Dialogue Benchmarking - Measures model helpfulness and quality using single-turn and multi-turn conversational benchmarks.
• Reinforcement Learning - Directly distills alignment preferences into language models.

Star history