thinking-machines-lab/tinker-cookbook

Features

• Language Model Fine-Tuning - Core framework for fine-tuning large language models using supervised and reinforcement learning.
• Large Language Model Fine-Tuning Frameworks - An open-source framework for fine-tuning large language models using supervised learning, reinforcement learning, and LoRA adapters.
• Reinforcement Learning Training - Training language models to maximize reward signals through on-policy rollouts, scoring, and importance-sampled loss updates.
• Supervised Training Pipelines - Uses configuration objects and dataset builders to set up and run supervised fine-tuning jobs.
• Reinforcement Learning Fine-Tuning - Applies the GRPO algorithm with reward functions to fine-tune models on math reasoning tasks.
• Large Language Model Fine-Tuning - Trains open-weight models from 1B to 1T+ parameters, including dense and mixture-of-experts architectures.
• Reinforcement Learning Environments - Defines custom ProblemEnv subclasses for RL training with token-level and message-level completion strategies.
• Low-Rank Adaptation - Performs parameter-efficient fine-tuning using LoRA adapters, matching full fine-tuning performance for many use cases.
• Multi-Stage Pipelines - Orchestrates supervised fine-tuning, preference model training, and reinforcement learning as sequential stages.
• GRPO Training Loops - Executes complete GRPO reinforcement learning training runs with higher-level abstractions.
• Parameter Efficient Fine-Tuning - Provides parameter-efficient fine-tuning via LoRA adapters, reducing memory and compute requirements.
• Preference Optimization - Runs direct preference optimization and full RLHF pipelines to align model outputs with human preferences.
• GRPO Training Loop Configurations - Implements full GRPO training loops with higher-level environment and dataset abstractions.
• Reinforcement Learning Training Utilities - Trains language models to maximize reward signals by sampling on-policy rollouts and applying importance-sampled loss.
• RLHF Training Pipelines - Provides a complete three-stage RLHF pipeline from supervised fine-tuning through preference optimization to reinforcement learning.
• Supervised Fine-Tuning - Ships a supervised fine-tuning loop that constructs data, performs forward and backward passes, and updates weights.
• Supervised Fine-Tuning Frameworks - Provides configuration objects and dataset builders for setting up supervised fine-tuning jobs.
• Training Recipes - Defines fine-tuning workflows as composable recipe files that chain data loading, model configuration, and training loops.
• Model Fine-Tuning - Provides supervised fine-tuning with cross-entropy loss and weight updates for language models.
• Vision Model Inputs - Supports passing image inputs alongside text for vision-language model fine-tuning and sampling.
• Direct Preference Optimization - Fine-tunes models using direct preference optimization from paired preference data without a separate reward model.
• Model Exporters - Merges a LoRA adapter into a full model and saves it in HuggingFace-compatible format.
• Model Uploads to Hub - Provides functionality to publish fine-tuned model checkpoints to the HuggingFace Hub.
• Chat Template Configurations - Tokenizes and formats prompts according to the chat template expected by different model families.
• Trained Model Outputs - Ships sampling and inference capabilities for generating outputs from fine-tuned models.
• Custom Environment Builders - Provides the ability to build custom reinforcement learning environments by subclassing ProblemEnv.
• Custom Loss Functions - Supports cross-entropy, importance sampling, PPO, and custom loss functions during model training.
• Hyperparameter Sweep Orchestrators - Includes a hyperparameter sweep engine that scans learning rates, LoRA ranks, and RL parameters to find optimal configurations.
• Fine-Tuned Model Evaluators - Runs evaluations on fine-tuned models to measure performance on standard benchmarks.
• Hub Weight Managers - Manages model checkpoints through save, load, download, and upload operations to a remote hub.
• Vision-Language Fine-Tunings - Trains multimodal models for image understanding tasks through the same fine-tuning API used for language models.
• Model Weight Management - Manages model checkpoints through save, load, download, and publish operations.
• Adapter Builders - Converts trained adapters into PEFT format for efficient serving and deployment.
• Adapter Exporters - Merges a LoRA adapter into a full model and exports it for use with the HuggingFace ecosystem.
• LoRA Adapter Builders and Exporters - Provides tools for building, merging, and exporting LoRA adapters for deployment and sharing.
• Multi-Adapter Serving Kernels - Converts trained adapters into PEFT format to serve the model with lower memory usage.
• Model-Specific Prompt Formats - Automatically applies model-specific chat templates to structure prompts before inference.
• Multi-Agent Training - Supports multi-agent reinforcement learning with self-play and competitive environments for training multiple agents.
• RL Reference Environments - Supports building custom RL environments by subclassing ProblemEnv for reinforcement learning tasks.
• Self-Play Training Pipelines - Sets up self-play and competitive environments for multi-agent reinforcement learning training.
• Async Request Throughput Optimizers - Optimizes throughput by sending multiple concurrent generation requests asynchronously.
• Model Evaluation and Benchmarking - Includes evaluation capabilities for running standard benchmarks on fine-tuned models.
• Model Checkpoint Uploads - Provides tools for uploading fine-tuned model checkpoints to the HuggingFace Hub for sharing.
• Checkpoint Saving and Restoration - Ships checkpoint saving and restoration utilities for managing model states during fine-tuning.
• Concurrent Sampling Engines - Ships an async concurrent sampling engine that maximizes throughput during training rollouts and evaluation.
• Concurrent Sampling and Training Pipelines - Pipelines concurrent sampling with forward-backward passes and optimizer steps for higher throughput.
• Asynchronous Request Handlers - Issues multiple concurrent API calls using futures to maximize throughput during training and inference.
• Concurrent AI Requesting - Implements an async concurrent sampling engine for parallel generation requests during training.

Star history