Projects sharing features with Aibrix

Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute manager designed to deploy and scale model workloads across diverse hardware architectures, including GPUs, CPUs, and heterogeneous hardware clusters. The platform enables model deployment across a wide range of targets, including NVIDIA GPUs, regional chipsets, and legacy hardware. It manages the execution of models across these varying environments to increase available computing capacity and optimize resource utilization. The system includes capabilities for distributed i

This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production via scalable network endpoints. It functions as a high-performance inference server, optimizer, and model lifecycle manager that handles model loading, request batching, and hardware acceleration. The system distinguishes itself through advanced orchestration and optimization capabilities, such as chaining multiple models into sequential workflows using execution graphs and employing dynamic batching to improve throughput and latency. It provides specialized support for generat

Seldon Core is a Kubernetes-based machine learning model server and MLOps inference framework. It functions as a multi-model serving engine and pipeline orchestrator, packaging models as scalable microservices that are exposed via standardized REST and gRPC APIs. The project distinguishes itself through graph-based inference pipelines that chain models and data transformers into sequential workflows. It optimizes hardware utilization via multi-model shared serving and dynamic memory overcommit strategies, while supporting production experimentation through weighted traffic routing, A/B testin

Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.

ClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and

Mooncake is a disaggregated large language model serving platform and distributed key-value store designed for high-performance inference infrastructure. It functions as a GPU memory orchestrator and KV cache management system that pools and transfers key-value caches across clusters to accelerate inference. The system distinguishes itself by separating the prefill and decode phases of inference into distinct hardware clusters to optimize resource utilization. It utilizes a high-performance RDMA distributed cache with zero-copy transfers to move data between compute nodes, bypassing the CPU t

KubeOperator is a comprehensive Kubernetes cluster management platform, infrastructure orchestrator, and multi-cluster manager. It functions as an enterprise Kubernetes distribution designed to automate the deployment, scaling, and lifecycle management of production clusters across diverse cloud platforms and physical machines. The platform distinguishes itself with specialized capabilities for air-gapped environments, including an offline installation engine that generates software archives and manages private registries for secure, non-internet deployments. It also provides a centralized da

Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

htop is a terminal-based system resource monitor and interactive process viewer. It functions as a text-user interface dashboard for overseeing hardware temperatures, load averages, and battery status while providing a comprehensive tool for monitoring and managing system processes. The application distinguishes itself through detailed process lifecycle management, allowing users to kill processes, adjust priorities via renicing, and assign CPU affinity to specific cores. It provides high-level visibility into system behavior through process hierarchy visualization and the ability to inspect

Beats is a collection of lightweight, modular agents designed to gather, process, and forward operational telemetry from distributed infrastructure to centralized storage and analysis platforms. These agents function as a distributed data transport layer, decoupling the collection of logs, metrics, and network events from their final delivery destination. By maintaining local state and managing data flow, the system ensures reliable transmission of information across heterogeneous environments. The project distinguishes itself through a modular pipeline architecture that allows for the assemb

Beszel is a self-hosted server monitoring platform designed to track real-time performance metrics across multiple host systems and containerized environments. It functions as a centralized dashboard that aggregates data on processor, memory, disk, and network usage, providing visibility into both host-level infrastructure and individual container workloads. The system utilizes lightweight agents to collect performance data, which is then streamed to a central hub and stored in a local relational database. It distinguishes itself through a real-time analytics engine that uses persistent bidir

Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat

Hyperf is a high-performance PHP coroutine framework designed for building microservices and middleware. It utilizes non-blocking coroutines to handle high concurrency and low-latency request processing, providing a foundation for scalable distributed systems. The framework is distinguished by an aspect-oriented programming based dependency injector that enables pluggable components and meta-programming. It includes a coroutine-optimized object-relational mapper with integrated model caching and an orchestration toolkit for microservice governance, featuring service discovery, circuit breaker

Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments. The framework distinguishes itself through a unified approach to reactive and imperative program

Odysseus is a self-hosted AI workspace and autonomous agent framework designed for deploying and managing large language models. It serves as a centralized platform for orchestrating agentic tasks, utilizing a model context protocol server to connect AI models to external system utilities, browser automation, and local hardware. The system distinguishes itself through a combination of retrieval-augmented generation and a RAG knowledge base, using vector stores and local embeddings to provide persistent semantic memory. It further integrates AI-driven communication management to triage email i

ClearML is a comprehensive MLOps platform designed to manage the entire machine learning lifecycle. It functions as an experiment tracking tool, a data versioning system, and a pipeline orchestrator, while providing infrastructure for GPU cluster management and model serving. The platform is distinguished by its ability to handle hybrid-cloud compute scheduling and fractional GPU allocation, allowing multiple workloads to share a single hardware accelerator. It employs a metadata-based approach to data versioning, using virtual views to track large datasets and artifacts without duplicating r

Spin is a WebAssembly serverless framework and development toolchain for building and running portable microservices. It functions as an event-driven orchestrator and runtime that executes WebAssembly components, allowing developers to map HTTP requests, Redis messages, and cron schedules to specific modules. The project distinguishes itself by implementing a Wasm-based AI inference gateway, enabling components to perform model inference and generate text embeddings. It utilizes the WebAssembly Component Model and WASI for language-agnostic composition and portable host interfacing, while emp

FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications. The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di

This project is a comprehensive infrastructure guide and technical reference for designing and deploying cloud native and AI native environments using Kubernetes. It serves as a manual for managing container orchestration, pod lifecycles, and declarative state reconciliation to maintain scalable cluster workloads. The resource provides instructional material on building custom controllers and implementing operational logic via the operator pattern. It also functions as a framework for optimizing the delivery of large language models through specialized gateways and workload scheduling. The h

mini-sglang is a collection of tools for large language model inference, serving as an OpenAI-compatible inference server, a memory-efficient prefill engine, and a tensor parallelism runtime. It also functions as a local batch processing engine for offline benchmarking and ablation studies. The project focuses on acceleration and memory management through a KV cache manager that reuses precomputed caches for shared request prefixes. It handles large model workloads by distributing tasks across multiple GPUs and manages peak memory consumption by splitting long input sequences into smaller chu

PaddleNLP is a development library and toolkit for training, fine-tuning, and deploying large and small language models using the PaddlePaddle framework. It provides a comprehensive suite for the entire natural language processing lifecycle, from model development to high-performance inference. The project features a standardized model zoo for loading and managing pre-trained models and tokenizers through a unified interface. It distinguishes itself with a specialized model compression framework that reduces memory footprints via weight precision conversion and lossless size optimization, alo

This project is a Kubernetes serverless framework and OCI container function platform. It provides a system for deploying event-driven functions and microservices as compatible container images onto a Kubernetes cluster. The platform includes an event-driven function orchestrator that triggers executions via HTTP requests or message streams. It features an auto-scaling function manager that adjusts the number of active instances based on real-time demand and scales down to zero during inactivity. A background queuing system is included to process asynchronous tasks and maintain application re

InternLM is a large language model and a comprehensive suite of weights designed for text generation and complex reasoning. It functions as an inference engine for serving responses, a fine-tuning framework for adjusting model weights, and a platform for building autonomous AI agents. The system is capable of processing long-context input sequences up to one million tokens for document analysis. It employs chain-of-thought reasoning to solve knowledge-intensive tasks by generating intermediate logic steps before producing a final answer. The project covers model weight optimization through s

lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models. It functions as a multi-modal model server and compression toolkit designed to serve models with high throughput and low latency. The system enables the distribution of model services across multiple machines using request-based load balancing and tensor parallelism. It includes specialized tools for model quantization and compression to reduce the memory footprint of weights and caches. The framework covers broad capability areas including production deployment, distributed