Awesome GitHub RepositoriesModel Serving

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke. The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequ

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

This is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.

Ce projet est un guide architectural MLOps et un framework pour concevoir et déployer des systèmes d'apprentissage profond dans des environnements de production. Il fournit une approche structurée pour le déploiement de l'inférence de modèles, l'orchestration de pipelines ML et la création d'architectures de machine learning de niveau production. Le projet se distingue par un accent mis sur l'apprentissage profond distribué et l'IA en périphérie (edge AI). Il couvre des méthodologies pour paralléliser l'entraînement des modèles sur plusieurs GPU afin de gérer de grands jeux de données et applique des techniques comme la quantification et la distillation pour réduire la taille des modèles pour le matériel embarqué. La surface de capacité s'étend à la surveillance et à l'observabilité, incluant le suivi de la performance des modèles, la dérive des données et les métriques d'expérience. Il aborde également l'orchestration des flux de travail de données, le versionnage des jeux de données via des magasins d'objets, et la gestion des requêtes d'inférence à haut volume en utilisant le traitement par lots adaptatif et l'orchestration basée sur des conteneurs.

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

gptme est un serveur et framework d'agent IA autonome conçu pour l'automatisation du système local, le développement logiciel et l'exécution de code. Il fonctionne comme un moteur d'exécution local qui permet aux modèles de langage d'exécuter des commandes shell, de modifier des fichiers locaux et d'interagir avec le système d'exploitation. Le projet fonctionne comme un client du protocole Model Context Protocol, s'intégrant avec des serveurs externes pour étendre les capacités de l'agent avec des outils et des sources de données standardisés. Il dispose d'un système de routage agnostique au fournisseur pour orchestrer les tâches à travers plusieurs API cloud propriétaires et backends IA locaux. Le système inclut des capacités pour l'automatisation de navigateur headless, l'analyse de contenu visuel et l'analyse de code basée sur les symboles pour mapper les bases de code. Pour garantir la sécurité, il implémente des garde-fous avec intervention humaine (human-in-the-loop) qui nécessitent une confirmation de l'utilisateur avant d'exécuter des changements système sensibles ou de finaliser des patchs de fichiers. L'application peut être déployée en tant que binaire de bureau autonome ou via une conteneurisation Docker.