Awesome GitHub RepositoriesModel Serving

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke. The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequ

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

This is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.

Acest proiect este un ghid arhitectural MLOps și un framework pentru proiectarea și implementarea sistemelor de deep learning în medii de producție. Oferă o abordare structurată pentru implementarea inferenței modelelor, orchestrarea pipeline-urilor ML și crearea de arhitecturi de machine learning la nivel de producție. Proiectul se distinge printr-un accent pe deep learning distribuit și optimizarea AI la margine (edge AI). Acoperă metodologii pentru paralelizarea antrenării modelelor pe mai multe GPU-uri pentru a gestiona seturi mari de date și aplică tehnici precum cuantizarea și distilarea pentru a reduce dimensiunea modelului pentru hardware-ul embedded. Suprafața de capabilități se extinde la monitorizare și observabilitate, incluzând urmărirea performanței modelului, data drift-ul și metricile experimentelor. Abordează, de asemenea, orchestrarea fluxului de lucru al datelor, versionarea seturilor de date prin object stores și gestionarea cererilor de inferență de mare volum folosind batching adaptiv și orchestrare bazată pe containere.

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

gptme este un server și framework de agenți AI autonomi conceput pentru automatizarea sistemelor locale, dezvoltarea software și execuția codului. Acesta funcționează ca un motor de execuție local care permite modelelor de limbaj să ruleze comenzi shell, să modifice fișiere locale și să interacționeze cu sistemul de operare. Proiectul funcționează ca un client Model Context Protocol, integrându-se cu servere externe pentru a extinde capabilitățile agentului cu instrumente și surse de date standardizate. Dispune de un sistem de rutare agnostic față de furnizor pentru a orchestra sarcinile prin mai multe API-uri cloud proprietare și backend-uri AI locale. Sistemul include capabilități pentru automatizarea browserului headless, analiza conținutului vizual și analiza codului bazată pe simboluri pentru maparea bazelor de cod. Pentru a asigura siguranța, implementează guardrail-uri human-in-the-loop care necesită confirmarea utilizatorului înainte de a executa modificări sensibile ale sistemului sau de a finaliza patch-urile de fișiere. Aplicația poate fi deployată ca un binar desktop standalone sau prin containerizare Docker.