arcee-ai/mergekit
Mergekit
MergeKit is a toolkit for combining multiple pre-trained large language models into a single entity using algorithmic blending. It provides a specialized system for parameter interpolation and weight extraction to unify model capabilities.
The project distinguishes itself through an evolutionary merge optimizer that tunes parameters based on quantitative evaluation metrics. It also features a mixture of experts orchestrator capable of converting dense models into sparse architectures and a tokenizer alignment tool for transplanting embeddings between different models.
The toolkit covers a broad range of merging operations, including layer-based model assembly, task vector arithmetic, and various weight interpolation methods such as spherical linear interpolation and linear averaging. It includes utilities for parameter pruning, low-rank weight extraction, and the orchestration of multi-stage merge workflows.
Support is provided for executing resource-intensive operations on remote cloud infrastructure and generating markdown model cards for hosting hubs.
Features
- Model Merging - Provides a comprehensive toolkit for combining multiple pretrained large language models into a single entity using algorithmic blending.
- Model Merging - Combines multiple pre-trained models into a single entity using algorithmic blending techniques to unify capabilities.
- Checkpoint Averaging - Combines multiple pre-trained models using linear weight averaging and task arithmetic.
- Evolutionary Hyperparameter Tuners - Tunes merge parameters automatically using evolutionary algorithms to maximize evaluation scores.
- Evolutionary Merge Optimization - Tunes merge parameters automatically using an evolutionary algorithm to maximize scores from evaluation metrics.
- Task Vector Mergings - Combines specific capabilities by calculating differences from a common base model and adding them together.
- Evolutionary Parameter Optimizers - Implements an evolutionary algorithm for tuning merge parameters based on quantitative evaluation metrics.
- Weighted Parameter Averaging - Computes a weighted average of parameters from multiple models to create a combined version.
- Mixture of Experts - Transforms multiple dense models into a sparse mixture-of-experts architecture by implementing a routing gating structure.
- Weight Approximation Extraction - Extracts low-rank approximations from fine-tuned models to produce compatible adapter weights.
- Layer-Wise Model Assembly - Provides capabilities to build hybrid model architectures by selecting and combining specific layer slices from different source checkpoints.
- Task-Vector Arithmetic - Isolates fine-tuned capabilities by calculating parameter differences from a base model and summing those vectors.
- Tokenizer Transplantation - Aligns and transplants tokenizer embeddings and vocabularies between models to enable speculative decoding or knowledge distillation.
- Tokenizer Vocabulary Merging - Aligns donor tokenizers with a base model's embedding space using approximation strategies to merge vocabularies.
- Weight Interpolation - Computes weighted linear and spherical interpolations of model parameters to blend capabilities from multiple pre-trained sources.
- Adaptive Parameter Prunings - Retains parameters with the largest magnitude changes to preserve significant modifications during a merge.
- Direct Checkpoint Manipulation - Implements merging algorithms directly on raw tensor files to reduce memory overhead during model assembly.
- Checkpoint Tensor Manipulation - Applies merging algorithms directly to raw PyTorch tensor files without requiring a model object instantiation.
- Merge Quality Benchmarking - Scores merged models using defined tasks and weighted metrics to determine the effectiveness of merge recipes.
- Variance Masking Merges - Weights model contributions by identifying and keeping parameter positions with the highest variance across vectors.
- Merge Pipelines - Defines sequences of merge operations where the output of one weight merge stage serves as the input for the next.
- Sparse Architectures - Constructs sparse architectures by combining attention and normalization from a base model with expert parameters.
- Dense-to-Sparse Conversions - Merges multiple dense models into a sparse architecture for deployment or further training.
- Salient Change Fusions - Combines models using importance scores and divergence metrics to merge key parameter differences.
- Magnitude-Based Parameter Pruning - Filters noise and interference by removing model parameters based on absolute change or variance across sources.
- Magnitude-Based Prunings - Removes the smallest and largest parameter changes to isolate mid-range modifications and filter noise.
- Model Sparsification - Implements parameter pruning and sign conflict resolution to create sparse model representations.
- Selective Parameter Interpolations - Interpolates models by increasing the influence of a secondary model where parameters are similar to the base.
- Interpolation Parameter Optimization - Calculates optimized weights for linear interpolation based on the geometric similarity between models.
- Spherical Linear Interpolations - Creates smooth transitions between models using spherical linear interpolation to maintain parameter magnitude.
- Multi-Stage Pipeline Processing - Orchestrates complex merge recipes as a sequence of interdependent processing steps where outputs feed into subsequent stages.
- Emerging Trends - Listed in the “Emerging Trends” section of the Llm Course awesome list.
- Model Utilities - Tools for merging multiple pretrained language models.
