How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc
StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The Stack v2 and some natural language text such as Wikipedia, Arxiv, and GitHub issues. The models use Grouped Query Attention, a context window of 16,384 tokens, with sliding window…
This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset
Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted
Run Mixtral-8x7B models in Colab or consumer desktops
The main features of dvmazur/mixtral-offloading are: Large Language Models, Large Language Models (LLMs).
Projects with overlapping indexed features include: bigscience-workshop/petals — Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer… chroma-core/chroma — Chroma is a specialized vector database designed to index and retrieve high-dimensional data representations for… berriai/litellm — LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model… bigcode-project/starcoder2 — StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The… artidoro/qlora — This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation… databrickslabs/dolly — Dolly is an instruction-tuned large language model designed to follow complex natural language directions. It operates…