Awesome GitHub RepositoriesModel Serving

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke. The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequ

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

This is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.

This project is an MLOps architectural guide and framework for designing and deploying deep learning systems into production environments. It provides a structured approach to model inference deployment, ML pipeline orchestration, and the creation of production-level machine learning architectures. The project distinguishes itself through a focus on distributed deep learning and edge AI optimization. It covers methodologies for parallelizing model training across multiple GPUs to handle large datasets and applies techniques like quantization and distillation to reduce model size for embedded

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

gptme هو خادم وإطار عمل لوكيل ذكاء اصطناعي مستقل مصمم لأتمتة النظام المحلي، وتطوير البرمجيات، وتنفيذ الكود. يعمل كمحرك تنفيذ محلي يتيح لنماذج اللغة تشغيل أوامر shell، وتعديل الملفات المحلية، والتفاعل مع نظام التشغيل. يعمل المشروع كعميل لبروتوكول سياق النموذج (Model Context Protocol)، ويتكامل مع خوادم خارجية لتوسيع قدرات الوكيل بأدوات ومصادر بيانات موحدة. يتميز بنظام توجيه مستقل عن المزود لتنسيق المهام عبر واجهات برمجة تطبيقات سحابية متعددة مملوكة وخلفيات ذكاء اصطناعي محلية. يتضمن النظام قدرات لأتمتة المتصفح بدون رأس، وتحليل المحتوى المرئي، وتحليل الكود القائم على الرموز لرسم خرائط قواعد الكود. لضمان السلامة، ينفذ حواجز حماية بشرية تتطلب تأكيد المستخدم قبل تنفيذ تغييرات النظام الحساسة أو وضع اللمسات الأخيرة على تصحيحات الملفات. يمكن نشر التطبيق كملف ثنائي مستقل لسطح المكتب أو عبر حاوية Docker.