Awesome GitHub RepositoriesModel Serving

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke. The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequ

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

This is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.

Dieses Projekt ist ein MLOps-Architekturleitfaden und ein Framework für das Design und Deployment von Deep-Learning-Systemen in Produktionsumgebungen. Es bietet einen strukturierten Ansatz für das Deployment von Modell-Inferenz, ML-Pipeline-Orchestrierung und die Erstellung von Machine-Learning-Architekturen auf Produktionsebene. Das Projekt zeichnet sich durch einen Fokus auf verteiltes Deep Learning und Edge-KI-Optimierung aus. Es deckt Methoden zur Parallelisierung des Modelltrainings über mehrere GPUs hinweg ab, um große Datensätze zu verarbeiten, und wendet Techniken wie Quantisierung und Destillation an, um die Modellgröße für Embedded-Hardware zu reduzieren. Die Funktionsfläche erstreckt sich auf Monitoring und Observability, einschließlich der Verfolgung von Modell-Performance, Data-Drift und Experiment-Metriken. Es adressiert zudem die Orchestrierung von Daten-Workflows, Datensatz-Versionierung über Object-Stores und die Verwaltung von Inferenzanfragen mit hohem Volumen mithilfe von adaptivem Batching und Container-basierter Orchestrierung.

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

gptme ist ein autonomer KI-Agenten-Server und ein Framework, das für lokale Systemautomatisierung, Softwareentwicklung und Codeausführung entwickelt wurde. Es arbeitet als lokale Ausführungs-Engine, die es Sprachmodellen ermöglicht, Shell-Befehle auszuführen, lokale Dateien zu ändern und mit dem Betriebssystem zu interagieren. Das Projekt fungiert als Client für das Model Context Protocol und integriert sich in externe Server, um Agentenfunktionen mit standardisierten Tools und Datenquellen zu erweitern. Es verfügt über ein providerunabhängiges Routing-System, um Aufgaben über mehrere proprietäre Cloud-APIs und lokale KI-Backends hinweg zu orchestrieren. Das System umfasst Funktionen für Headless-Browser-Automatisierung, visuelle Inhaltsanalyse und symbolbasierte Codeanalyse zur Kartierung von Codebasen. Um Sicherheit zu gewährleisten, implementiert es Human-in-the-Loop-Leitplanken, die eine Benutzerbestätigung erfordern, bevor sensible Systemänderungen ausgeführt oder Datei-Patches finalisiert werden. Die Anwendung kann als eigenständige Desktop-Binärdatei oder über Docker-Containerisierung bereitgestellt werden.