awesome-repositories.com
博客
awesome-repositories.com

通过 AI 驱动的搜索，发现最优秀的开源仓库。

探索精选搜索开源替代品自托管软件博客网站地图
项目关于排名机制媒体报道MCP 服务器
法律隐私政策服务条款
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
eric-mitchell avatar

eric-mitchell/direct-preference-optimization

0
View on GitHub↗
2,888 星标·236 分支·Python·Apache-2.0·3 次浏览

Direct Preference Optimization

该项目是一个用于将大语言模型与人类偏好对齐的框架。它提供了一个库，通过将偏好数据直接映射到策略目标来优化模型行为，从而无需单独的奖励模型。

该框架利用闭式数学目标，通过最大化偏好响应相对于非偏好响应的似然度来调整模型权重。它包含一个监督微调工具包，允许用户在应用偏好优化技术之前，在演示数据上建立稳定的基准。

该库支持分布式训练，通过数据分片和并行化策略，实现跨多个硬件加速器的模型对齐扩展。它还提供了准备自定义偏好数据集的实用程序，将提示词和响应对映射为结构化的人类反馈，以供训练流水线使用。

Features

  • Direct Preference Optimization - Optimizes language models using human feedback data through direct preference optimization algorithms.
  • Preference Alignment Objectives - Adjusts model weights by maximizing the likelihood of preferred responses relative to dispreferred ones using a closed-form mathematical objective.
  • Reward Modeling - Optimizes language models by mapping preference data directly to a policy objective without requiring a separate reward model.
  • Fine-Tuning Toolkits - Provides a collection of training utilities for aligning language models on specific datasets.
  • Gradient-Based Parameter Updates - Updates model parameters by calculating divergence between preferred and rejected outputs to shift probability distributions.
  • Data Parallelism - Splits training datasets across multiple devices and synchronizes gradients to accelerate model alignment.
  • Large Language Models - Optimizes language models using human preference data to ensure generated outputs match desired behaviors.
  • Large-Scale Model Training - Scales the training process across multiple hardware accelerators to reduce computation time for large models.
  • Distributed Training - Scales model training across multiple hardware accelerators using parallelization strategies like data sharding.
  • Large Language Model Training Frameworks - Provides a framework for scaling the training of large language models across multiple hardware accelerators.
  • Supervised Fine-Tuning Workflows - Trains language models on curated datasets to establish a strong baseline performance before applying advanced optimization techniques.
  • Supervised Fine-Tuning - Establishes a stable baseline by training the model on high-quality demonstration data before applying preference-based fine-tuning.
  • Model Fine-Tuning - Trains models on specific datasets to ensure policy data remains in-distribution before applying preference-based learning.

Star 历史

eric-mitchell/direct-preference-optimization 的 Star 历史图表eric-mitchell/direct-preference-optimization 的 Star 历史图表

AI 搜索

探索更多 awesome 仓库

用简单的语言描述您的需求 —— AI 将根据相关性为您从数千个精选开源项目中进行排序。

Start searching with AI

包含 Direct Preference Optimization 的精选搜索

收录 Direct Preference Optimization 的精选合集。
  • Prompt 优化与自动调优框架
  • 大模型对齐与 RLHF 框架

Direct Preference Optimization 的开源替代方案

相似的开源项目，按与 Direct Preference Optimization 的功能重合度排序。
  • internlm/xtunerInternLM 的头像

    InternLM/xtuner

    5,150在 GitHub 上查看↗

    xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised fine-tuning, and the optimization of vision-language multimodal models. It serves as a distributed training accelerator and a specialized framework for scaling Mixture-of-Experts models and aligning model behavior through reinforcement learning from human feedback. The project distinguishes itself through advanced memory and compute optimizations, such as sequence parallelism for ultra-long context windows and interleaved pipeline parallelism to reduce GPU idle time. It provide

    Pythonagentdeepseek-v3gpt-oss
    在 GitHub 上查看↗5,150
  • allenai/open-instructallenai 的头像

    allenai/open-instruct

    3,586在 GitHub 上查看↗

    Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt

    Python
    在 GitHub 上查看↗3,586
  • huggingface/alignment-handbookhuggingface 的头像

    huggingface/alignment-handbook

    5,621在 GitHub 上查看↗

    This project is an alignment framework and suite of pipelines for training language models using supervised fine-tuning and preference optimization. It provides tools for executing large-scale distributed training across multiple GPUs and compute nodes, alongside a system for measuring model helpfulness and dialogue quality through single-turn and multi-turn benchmarks. The framework includes specialized tools for direct preference optimization to refine model behavior using paired data without a separate reward model. It also supports constitutional AI alignment and the training of reward mo

    Python
    在 GitHub 上查看↗5,621
  • nndl/llm-beginnernndl 的头像

    nndl/llm-beginner

    6,421在 GitHub 上查看↗

    This project is a collection of educational resources and technical guides focused on the development and implementation of large language models. It provides a comprehensive curriculum covering transformer architectures, training methods, and deployment strategies. The materials provide detailed instructions for building autonomous agents using reasoning loops and tool integration, as well as guides for fine-tuning models through supervised learning and preference optimization. It also includes tutorials for constructing retrieval augmented generation pipelines and implementing transformer m

    Pythonagentfudannlpllm
    在 GitHub 上查看↗6,421
查看 Direct Preference Optimization 的所有 30 个替代方案→

常见问题解答

eric-mitchell/direct-preference-optimization 是做什么的？

该项目是一个用于将大语言模型与人类偏好对齐的框架。它提供了一个库，通过将偏好数据直接映射到策略目标来优化模型行为，从而无需单独的奖励模型。

eric-mitchell/direct-preference-optimization 的主要功能有哪些？

eric-mitchell/direct-preference-optimization 的主要功能包括：Direct Preference Optimization, Preference Alignment Objectives, Reward Modeling, Fine-Tuning Toolkits, Gradient-Based Parameter Updates, Data Parallelism, Large Language Models, Large-Scale Model Training。

eric-mitchell/direct-preference-optimization 有哪些开源替代品？

eric-mitchell/direct-preference-optimization 的开源替代品包括： internlm/xtuner — xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised… allenai/open-instruct — Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a… huggingface/alignment-handbook — This project is an alignment framework and suite of pipelines for training language models using supervised… nndl/llm-beginner — This project is a collection of educational resources and technical guides focused on the development and… eleutherai/gpt-neox — gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It… inclusionai/areal — AReaL is a system for agent orchestration, distributed model training, and parameter-efficient tuning. It provides a…