Projects sharing features with LMCache

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc

FusionCache is a .NET caching framework that manages local and distributed data storage. It functions as a hybrid cache implementation, combining fast in-memory storage with a distributed second-level cache to reduce latency and remote data fetches. The framework includes a cache stampede protector that ensures only one request fetches missing data while concurrent calls wait. It synchronizes data across multiple application nodes using a backplane and provides an observability-driven approach with built-in metrics, structured logging, and telemetry. The system covers a broad range of cachin

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

FoundationDB is an ACID-compliant distributed transactional key-value store. It functions as a scalable database engine that ensures strict serializability and data consistency across a cluster of servers using a shared-nothing architecture. The system is distinguished by its multi-region replication capabilities, allowing data to be synchronized across different datacenters for high availability and disaster recovery. It utilizes optimistic concurrency control to manage distributed transactions and employs a majority-based coordination system to maintain cluster state. The platform provides

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

Dice is a low-latency key-value store based on Valkey that functions as a hierarchical storage engine. It combines a high-speed memory store with a tiered cache system to manage data across multiple storage levels. The system extends memory capacity by implementing a disk-backed cache that automatically persists evicted items to disk and restores them during cache misses. This storage tiering balances retrieval speed and operational cost by distributing data between fast memory and slower disk storage. The engine includes a query subscription service that monitors specific data patterns and

DiceDB is a distributed cache manager and hierarchical storage engine based on a Valkey key-value store. It functions as a high-availability key-value cluster that balances performance and cost by distributing data across memory and disk tiers. The system supports working sets larger than available RAM by persisting evicted keys to disk and restoring them on demand. It also includes a real-time query subscription system that notifies clients when data matching specific criteria is updated. The project provides distributed cache coordination through master node health monitoring and quorum-ba

This project is a collection of reference implementations and practical guides for building enterprise Java applications using the Spring Boot framework. It serves as a backend project gallery and implementation guide, providing a set of architectures for common server patterns. The repository distinguishes itself through a focus on distributed system design, offering examples for global unique identifier generation, distributed caching, and full-text search. It also includes templates and examples for creating custom Spring Boot starters to encapsulate shared dependencies and configurations

KeyDB is a multithreaded in-memory key-value store and distributed cache. It functions as a NoSQL database utilizing multi-version concurrency control to execute non-blocking queries and scans. The project is a multithreaded fork of Redis that maintains protocol compatibility while utilizing a multithreaded architecture to scale across multi-core hardware. It distinguishes itself with flash-tiered storage, allowing the system to offload data from primary RAM to SSD or flash storage to increase total capacity. The system supports high availability through active-active mesh replication and mu

Ignite is a distributed in-memory data grid and compute platform. It functions as a distributed SQL database and storage engine designed to store and process large datasets in RAM to minimize latency and increase calculation speed. The system is distinguished by a multi-tier storage engine that manages data placement across memory and disk to balance high-speed access with large capacity. It features a distributed compute grid that executes custom logic directly on the nodes where data resides to reduce network traffic. The platform provides a broad set of capabilities including ACID transac

This is a PHP library providing standardized interfaces for caching data through the PSR-6 and PSR-16 specifications. It functions as a cache pool manager and distributed cache adapter, allowing applications to organize key-value pairs into logical repositories and namespaces. The project includes a dedicated cache stampede prevention tool that protects systems from CPU spikes during value expiration. This is achieved through locking mechanisms for concurrent requests and probabilistic early expiration. The library supports object serialization with optional encryption and compression to tra

ERNIE is a development toolkit for training, fine-tuning, and deploying large language models built on the PaddlePaddle deep learning platform. It provides a comprehensive suite of core components, including an inference server for vision and language models, a training and fine-tuning toolkit, and a framework for building retrieval-augmented generation systems using private knowledge bases. The project features multimodal AI models capable of reasoning across text, images, and video to perform complex visual understanding and information extraction. It distinguishes itself through specialize

Athens is a Go module proxy server and dependency cache that provides a persistent storage system for Go dependencies. It acts as a mirror and datastore to ensure reproducible build environments by storing immutable copies of external packages, protecting against upstream deletions or outages. The project distinguishes itself by serving as a secure gateway for private Go module hosting, utilizing authentication tokens, SSH keys, and GitHub Apps to retrieve dependencies from private version control systems. It further enables software dependency compliance through request filtering and checksu

Garnet is a multi-threaded in-memory database and distributed key-value store. It functions as a high-performance remote cache store that implements the RESP wire protocol to maintain compatibility with existing Redis clients and libraries. The project is distinguished by a shared-memory architecture that enables parallel request processing across multiple cores for sub-millisecond latency. It features a tiered storage system that automatically offloads colder data from system memory to SSD or cloud storage layers, and includes a specialized vector search database for high-dimensional similar

MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp

3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference workloads. It functions as a strongly consistent storage layer that utilizes a disaggregated architecture to pool SSDs and memory resources across multiple nodes. The system provides specialized storage implementations including an AI training checkpoint store for parallel state preservation and a distributed key-value cache store for decoder layer vectors to optimize inference processing. It ensures data integrity through chain replication and apportioned query distribution. The

mini-sglang is a collection of tools for large language model inference, serving as an OpenAI-compatible inference server, a memory-efficient prefill engine, and a tensor parallelism runtime. It also functions as a local batch processing engine for offline benchmarking and ablation studies. The project focuses on acceleration and memory management through a KV cache manager that reuses precomputed caches for shared request prefixes. It handles large model workloads by distributing tasks across multiple GPUs and manages peak memory consumption by splitting long input sequences into smaller chu

Olric is a distributed data grid and in-memory key-value store that partitions and replicates data across a cluster of servers. It serves as a shared memory system for managing distributed maps, performing atomic operations, and acting as an in-memory data cache. The system provides a distributed locking mechanism for concurrency control and a pub-sub messaging system that broadcasts and routes messages over named channels across the cluster. The platform covers wide-ranging capabilities including cluster management and orchestration, data replication with configurable quorums, and automated

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This