Awesome GitHub RepositoriesServing Frameworks

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode

OpenLLM is a framework for deploying, managing, and scaling open-source large language models

Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

ClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and

LitServe ist ein Python-Framework für KI-Inferenzserver und LLM-Serving, das für Inferenz mit hoher Nebenläufigkeit konzipiert ist. Es fungiert als verteilter KI-Modellserver und Inferenz-Engine mit dynamischem Batching und bietet die Werkzeuge zum Erstellen und Hosten benutzerdefinierter Server, auf denen KI-Modelle laufen. Das Framework zeichnet sich durch eine Request-Queue mit dynamischem Batching aus, die einzelne Inferenzanfragen zu einzelnen Tensoren gruppiert, um den GPU-Durchsatz zu maximieren. Es unterstützt verteiltes GPU-Skalieren, wodurch Modell-Workloads auf mehrere Hardware-Beschleuniger verteilt werden können, um Rechenlasten auszugleichen und die Gesamtkapazität zu erhöhen. Das System bietet eine High-Level-Wrapper-Schnittstelle, die Request-Preprocessing und Postprocessing von der Kern-Modellausführungslogik entkoppelt. Es enthält zudem Funktionen für Echtzeit-Modell-Streaming zur inkrementellen Bereitstellung von Ausgaben und nutzt eine asynchrone Event-Loop zur Handhabung gleichzeitiger Netzwerkanfragen.