arcee-ai/mergekit

Features

• Model Merging - Provides a comprehensive toolkit for combining multiple pretrained large language models into a single entity using algorithmic blending.
• Model Merging - Combines multiple pre-trained models into a single entity using algorithmic blending techniques to unify capabilities.
• Checkpoint Averaging - Combines multiple pre-trained models using linear weight averaging and task arithmetic.
• Evolutionary Hyperparameter Tuners - Tunes merge parameters automatically using evolutionary algorithms to maximize evaluation scores.
• Evolutionary Merge Optimization - Tunes merge parameters automatically using an evolutionary algorithm to maximize scores from evaluation metrics.
• Task Vector Mergings - Combines specific capabilities by calculating differences from a common base model and adding them together.
• Evolutionary Parameter Optimizers - Implements an evolutionary algorithm for tuning merge parameters based on quantitative evaluation metrics.
• Weighted Parameter Averaging - Computes a weighted average of parameters from multiple models to create a combined version.
• Mixture of Experts - Transforms multiple dense models into a sparse mixture-of-experts architecture by implementing a routing gating structure.
• Weight Approximation Extraction - Extracts low-rank approximations from fine-tuned models to produce compatible adapter weights.
• Layer-Wise Model Assembly - Provides capabilities to build hybrid model architectures by selecting and combining specific layer slices from different source checkpoints.
• Task-Vector Arithmetic - Isolates fine-tuned capabilities by calculating parameter differences from a base model and summing those vectors.
• Tokenizer Transplantation - Aligns and transplants tokenizer embeddings and vocabularies between models to enable speculative decoding or knowledge distillation.
• Tokenizer Vocabulary Merging - Aligns donor tokenizers with a base model's embedding space using approximation strategies to merge vocabularies.
• Weight Interpolation - Computes weighted linear and spherical interpolations of model parameters to blend capabilities from multiple pre-trained sources.
• Adaptive Parameter Prunings - Retains parameters with the largest magnitude changes to preserve significant modifications during a merge.
• Direct Checkpoint Manipulation - Implements merging algorithms directly on raw tensor files to reduce memory overhead during model assembly.
• Checkpoint Tensor Manipulation - Applies merging algorithms directly to raw PyTorch tensor files without requiring a model object instantiation.
• Merge Quality Benchmarking - Scores merged models using defined tasks and weighted metrics to determine the effectiveness of merge recipes.
• Variance Masking Merges - Weights model contributions by identifying and keeping parameter positions with the highest variance across vectors.
• Merge Pipelines - Defines sequences of merge operations where the output of one weight merge stage serves as the input for the next.
• Sparse Architectures - Constructs sparse architectures by combining attention and normalization from a base model with expert parameters.
• Dense-to-Sparse Conversions - Merges multiple dense models into a sparse architecture for deployment or further training.
• Salient Change Fusions - Combines models using importance scores and divergence metrics to merge key parameter differences.
• Magnitude-Based Parameter Pruning - Filters noise and interference by removing model parameters based on absolute change or variance across sources.
• Magnitude-Based Prunings - Removes the smallest and largest parameter changes to isolate mid-range modifications and filter noise.
• Model Sparsification - Implements parameter pruning and sign conflict resolution to create sparse model representations.
• Selective Parameter Interpolations - Interpolates models by increasing the influence of a secondary model where parameters are similar to the base.
• Interpolation Parameter Optimization - Calculates optimized weights for linear interpolation based on the geometric similarity between models.
• Spherical Linear Interpolations - Creates smooth transitions between models using spherical linear interpolation to maintain parameter magnitude.
• Multi-Stage Pipeline Processing - Orchestrates complex merge recipes as a sequence of interdependent processing steps where outputs feed into subsequent stages.
• Emerging Trends - Listed in the “Emerging Trends” section of the Llm Course awesome list.
• Model Utilities - Tools for merging multiple pretrained language models.

Star history