eric-mitchell/direct-preference-optimization
Direct Preference Optimization
This project is a framework for aligning large language models with human preferences. It provides a library for optimizing model behavior by mapping preference data directly to a policy objective, bypassing the need for a separate reward model.
The framework utilizes a closed-form mathematical objective to adjust model weights by maximizing the likelihood of preferred responses relative to dispreferred ones. It includes a toolkit for supervised fine-tuning, allowing users to establish a stable baseline on demonstration data before applying preference-based optimization techniques.
The library supports distributed training, enabling the scaling of model alignment across multiple hardware accelerators through data sharding and parallelization strategies. It also provides utilities for preparing custom preference datasets, mapping prompts and response pairs to structure human feedback for the training pipeline.
