Awesome GitHub RepositoriesServing Frameworks

Jina is a cloud-native framework for building and deploying multimodal AI applications that process text, images, and audio across distributed microservices. It functions as an inference orchestrator and a distributed model gateway, providing a containerized stack to organize AI executors into operational pipelines. The system manages large language model workloads through token-streamed response delivery and dynamic batching to increase hardware throughput. It utilizes a protocol-agnostic communication layer to route data across different machine learning frameworks. The framework covers hi

Triton Inference Server is a high-performance server designed to deploy machine learning models from multiple frameworks across GPUs and CPUs. It functions as a hardware-accelerated inference engine and a gRPC inference gateway, providing a standardized communication layer for transmitting binary tensor data with low latency. The system acts as a multi-framework model orchestrator, allowing users to link multiple AI models into ensembles and scripts to create complex inference pipelines. It also serves as a model lifecycle manager, providing controls to load, unload, and monitor the performan

This project is a platform for the deployment of open source large language and multimodal models. It provides a unified interface to serve text, image, and speech models across local or cloud hardware. The system enables distributed AI inference by orchestrating model workloads across multiple nodes and devices. It includes a unified API adapter layer to standardize inputs and outputs, as well as tools for multimodal chat and structural image generation. The platform covers a broad capability surface including request batching for throughput optimization, dynamic model loading, and integrat

BentoML is a machine learning model serving framework and GPU-accelerated inference server designed to package, deploy, and scale AI models as production-ready REST APIs. It functions as an AI model lifecycle manager and an inference graph orchestrator, enabling the chaining of multiple models and custom logic into complex pipelines for advanced task sequences. The framework distinguishes itself through a dynamic batching engine that optimizes GPU throughput and an artifact-based packaging system that bundles model weights and dependencies into immutable archives for consistent deployment. It

TensorFlow Serving is a high-performance machine learning inference server designed to deploy TensorFlow models to production environments. It functions as a complete serving system that executes predictions on input data through a graph executor, providing network endpoints that eliminate the need for a separate runtime environment for client applications. The system is distinguished by its model version manager, which organizes and selects specific model versions within a directory hierarchy. It uses a filesystem watcher to detect new model versions and trigger automatic updates without int

Dieses Projekt ist ein PyTorch-Framework für das Model-Serving, das darauf ausgelegt ist, Machine-Learning-Modelle in der Produktion über skalierbare Netzwerk-Endpunkte bereitzustellen. Es fungiert als leistungsstarker Inference-Server, Optimierer und Modell-Lifecycle-Manager, der das Laden von Modellen, Request-Batching und Hardware-Beschleunigung übernimmt. Das System zeichnet sich durch fortschrittliche Orchestrierungs- und Optimierungsfunktionen aus, wie etwa das Verketten mehrerer Modelle zu sequenziellen Workflows mittels Ausführungsgraphen und den Einsatz von Dynamic Batching zur Verbesserung von Durchsatz und Latenz. Es bietet spezialisierte Unterstützung für generative KI und Large Language Models durch Continuous Batching und Tensor-Parallelität. Zu den breiten Funktionsbereichen gehören GPU-Ressourcenmanagement für diverse Hardware wie NVIDIA, AMD und Apple Silicon sowie ein umfassendes Lifecycle-Management für Registrierung, Versionierung und Worker-Skalierung. Zudem integriert es Observability-Tools zur Überwachung des Systemzustands und der Modellleistung über Prometheus-kompatible Metriken. Der Server wird über eine Kommandozeilenschnittstelle verwaltet, die zur Steuerung des Lifecycles und zur Konfiguration von Laufzeitparametern dient.

⚡ Langchain apps in production using Jina & FastAPI

RayLLM - LLMs on Ray (Archived). Read README for more info.

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

The web framework for building LLM microservices deprecated

Kubernetes AI Toolchain Operator

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Supercharge AI Agents, Safely