LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc
StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The Stack v2 and some natural language text such as Wikipedia, Arxiv, and GitHub issues. The models use Grouped Query Attention, a context window of 16,384 tokens, with sliding window…
This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset
Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted
Run Mixtral-8x7B models in Colab or consumer desktops
Principalele funcționalități ale dvmazur/mixtral-offloading sunt: Large Language Models, Large Language Models (LLMs).
Alternativele open-source pentru dvmazur/mixtral-offloading includ: bigscience-workshop/petals — Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer… chroma-core/chroma — Chroma is a specialized vector database designed to index and retrieve high-dimensional data representations for… berriai/litellm — LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model… bigcode-project/starcoder2 — StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The… artidoro/qlora — This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation… databrickslabs/dolly — Dolly is an instruction-tuned large language model designed to follow complex natural language directions. It operates…