Direct Preference Optimization

Dieses Projekt ist ein Framework zur Ausrichtung großer Sprachmodelle an menschlichen Präferenzen. Es bietet eine Bibliothek zur Optimierung des Modellverhaltens durch die direkte Abbildung von Präferenzdaten auf ein Policy-Ziel, wodurch die Notwendigkeit eines separaten Belohnungsmodells entfällt.

Das Framework nutzt ein mathematisches Closed-Form-Ziel, um Modellgewichte durch Maximierung der Wahrscheinlichkeit bevorzugter Antworten gegenüber nicht bevorzugten anzupassen. Es enthält ein Toolkit für überwachtes Fine-Tuning, das es Benutzern ermöglicht, eine stabile Basis auf Demonstrationsdaten zu etablieren, bevor präferenzbasierte Optimierungstechniken angewendet werden.

Die Bibliothek unterstützt verteiltes Training, was die Skalierung der Modellausrichtung über mehrere Hardware-Beschleuniger hinweg durch Data-Sharding- und Parallelisierungsstrategien ermöglicht. Sie bietet zudem Dienstprogramme zur Vorbereitung benutzerdefinierter Präferenzdatensätze, die Prompts und Antwortpaare abbilden, um menschliches Feedback für die Trainingspipeline zu strukturieren.

Features

Direct Preference Optimization - Optimizes language models using human feedback data through direct preference optimization algorithms.
Preference Alignment Objectives - Adjusts model weights by maximizing the likelihood of preferred responses relative to dispreferred ones using a closed-form mathematical objective.
Reward Modeling - Optimizes language models by mapping preference data directly to a policy objective without requiring a separate reward model.
Fine-Tuning Toolkits - Provides a collection of training utilities for aligning language models on specific datasets.
Gradient-Based Parameter Updates - Updates model parameters by calculating divergence between preferred and rejected outputs to shift probability distributions.
Data Parallelism - Splits training datasets across multiple devices and synchronizes gradients to accelerate model alignment.
Large Language Models - Optimizes language models using human preference data to ensure generated outputs match desired behaviors.
Large-Scale Model Training - Scales the training process across multiple hardware accelerators to reduce computation time for large models.
Distributed Training - Scales model training across multiple hardware accelerators using parallelization strategies like data sharding.
Large Language Model Training Frameworks - Provides a framework for scaling the training of large language models across multiple hardware accelerators.
Supervised Fine-Tuning Workflows - Trains language models on curated datasets to establish a strong baseline performance before applying advanced optimization techniques.
Supervised Fine-Tuning - Establishes a stable baseline by training the model on high-quality demonstration data before applying preference-based fine-tuning.
Model Fine-Tuning - Trains models on specific datasets to ensure policy data remains in-distribution before applying preference-based learning.

Star-Verlauf

eric-mitchelldirect-preference-optimization

Name: eric-mitchell/direct-preference-optimization
Author: eric-mitchell

View on GitHub

2,888 Stars236 ForksPythonApache-2.03 Aufrufe

Direct Preference Optimization

Features

Direct Preference Optimization - Optimizes language models using human feedback data through direct preference optimization algorithms.
Preference Alignment Objectives - Adjusts model weights by maximizing the likelihood of preferred responses relative to dispreferred ones using a closed-form mathematical objective.
Reward Modeling - Optimizes language models by mapping preference data directly to a policy objective without requiring a separate reward model.
Fine-Tuning Toolkits - Provides a collection of training utilities for aligning language models on specific datasets.
Gradient-Based Parameter Updates - Updates model parameters by calculating divergence between preferred and rejected outputs to shift probability distributions.
Data Parallelism - Splits training datasets across multiple devices and synchronizes gradients to accelerate model alignment.
Large Language Models - Optimizes language models using human preference data to ensure generated outputs match desired behaviors.
Large-Scale Model Training - Scales the training process across multiple hardware accelerators to reduce computation time for large models.
Distributed Training - Scales model training across multiple hardware accelerators using parallelization strategies like data sharding.
Large Language Model Training Frameworks - Provides a framework for scaling the training of large language models across multiple hardware accelerators.
Supervised Fine-Tuning Workflows - Trains language models on curated datasets to establish a strong baseline performance before applying advanced optimization techniques.
Supervised Fine-Tuning - Establishes a stable baseline by training the model on high-quality demonstration data before applying preference-based fine-tuning.
Model Fine-Tuning - Trains models on specific datasets to ensure policy data remains in-distribution before applying preference-based learning.

Open-Source-Alternativen zu Direct Preference Optimization

Ähnliche Open-Source-Projekte, sortiert nach der Anzahl der gemeinsamen Funktionen mit Direct Preference Optimization.

internlm/xtuner
InternLM/xtuner
5,150Auf GitHub ansehen
xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised fine-tuning, and the optimization of vision-language multimodal models. It serves as a distributed training accelerator and a specialized framework for scaling Mixture-of-Experts models and aligning model behavior through reinforcement learning from human feedback. The project distinguishes itself through advanced memory and compute optimizations, such as sequence parallelism for ultra-long context windows and interleaved pipeline parallelism to reduce GPU idle time. It provide
Pythonagentdeepseek-v3gpt-oss
Auf GitHub ansehen5,150
allenai/open-instruct
allenai/open-instruct
3,586Auf GitHub ansehen
Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt
Python
Auf GitHub ansehen3,586
huggingface/alignment-handbook
huggingface/alignment-handbook
5,621Auf GitHub ansehen
This project is an alignment framework and suite of pipelines for training language models using supervised fine-tuning and preference optimization. It provides tools for executing large-scale distributed training across multiple GPUs and compute nodes, alongside a system for measuring model helpfulness and dialogue quality through single-turn and multi-turn benchmarks. The framework includes specialized tools for direct preference optimization to refine model behavior using paired data without a separate reward model. It also supports constitutional AI alignment and the training of reward mo
Python
Auf GitHub ansehen5,621
nndl/llm-beginner
nndl/llm-beginner
6,421Auf GitHub ansehen
This project is a collection of educational resources and technical guides focused on the development and implementation of large language models. It provides a comprehensive curriculum covering transformer architectures, training methods, and deployment strategies. The materials provide detailed instructions for building autonomous agents using reasoning loops and tool integration, as well as guides for fine-tuning models through supervised learning and preference optimization. It also includes tutorials for constructing retrieval augmented generation pipelines and implementing transformer m
Pythonagentfudannlpllm
Auf GitHub ansehen6,421

Alle 30 Alternativen zu Direct Preference Optimization anzeigen

Kuratierte Suchen mit Direct Preference Optimization

Handverlesene Sammlungen, in denen Direct Preference Optimization vorkommt.

Häufig gestellte Fragen

Was macht eric-mitchell/direct-preference-optimization?

Was sind die Hauptfunktionen von eric-mitchell/direct-preference-optimization?

Die Hauptfunktionen von eric-mitchell/direct-preference-optimization sind: Direct Preference Optimization, Preference Alignment Objectives, Reward Modeling, Fine-Tuning Toolkits, Gradient-Based Parameter Updates, Data Parallelism, Large Language Models, Large-Scale Model Training.

Welche Open-Source-Alternativen gibt es zu eric-mitchell/direct-preference-optimization?

Open-Source-Alternativen zu eric-mitchell/direct-preference-optimization sind unter anderem: internlm/xtuner — xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised… allenai/open-instruct — Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a… huggingface/alignment-handbook — This project is an alignment framework and suite of pipelines for training language models using supervised… nndl/llm-beginner — This project is a collection of educational resources and technical guides focused on the development and… eleutherai/gpt-neox — gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It… inclusionai/areal — AReaL is a system for agent orchestration, distributed model training, and parameter-efficient tuning. It provides a…

Direct Preference Optimization

Features

Star-Verlauf

Direct Preference Optimization

Features

Open-Source-Alternativen zu Direct Preference Optimization

InternLM/xtuner

allenai/open-instruct

huggingface/alignment-handbook

nndl/llm-beginner

Kuratierte Suchen mit Direct Preference Optimization

Häufig gestellte Fragen

Star-Verlauf

Open-Source-Alternativen zu Direct Preference Optimization

InternLM/xtuner

allenai/open-instruct

huggingface/alignment-handbook

nndl/llm-beginner

Häufig gestellte Fragen

Kuratierte Suchen mit Direct Preference Optimization