Awesome GitHub RepositoriesModel Serving

DeepPavlov is a conversational AI framework and deep learning NLP library designed for building end-to-end dialogue systems and chatbots. It functions as an NLP pipeline orchestrator that allows users to compose pre-trained models and text processing components into sequential data flows for complex linguistic tasks. The system is distinguished by its ability to act as a chatbot deployment server, exposing trained conversational models as web services via REST and Socket APIs. It utilizes JSON-based pipeline configurations and dynamic variable interpolation to decouple model logic from infras

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke. The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequ

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

This is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.

Este proyecto es una guía arquitectónica de MLOps y framework para diseñar y desplegar sistemas de deep learning en entornos de producción. Proporciona un enfoque estructurado para el despliegue de inferencia de modelos, orquestación de pipelines de ML y la creación de arquitecturas de machine learning a nivel de producción. El proyecto se distingue por un enfoque en deep learning distribuido y edge AI. Cubre metodologías para paralelizar el entrenamiento de modelos a través de múltiples GPUs para manejar grandes datasets y aplica técnicas como cuantización y destilación para reducir el tamaño del modelo para hardware embebido. La superficie de capacidad se extiende al monitoreo y observabilidad, incluyendo el seguimiento del rendimiento del modelo, deriva de datos (data drift) y métricas de experimentos. También aborda la orquestación de flujos de trabajo de datos, versionado de datasets mediante almacenes de objetos y la gestión de solicitudes de inferencia de alto volumen utilizando procesamiento por lotes adaptativo y orquestación basada en contenedores.

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

gptme es un servidor y framework de agentes de IA autónomos diseñado para la automatización de sistemas locales, desarrollo de software y ejecución de código. Opera como un motor de ejecución local que permite a los modelos de lenguaje ejecutar comandos de shell, modificar archivos locales e interactuar con el sistema operativo. El proyecto funciona como un cliente del Protocolo de Contexto de Modelo (Model Context Protocol), integrándose con servidores externos para ampliar las capacidades del agente con herramientas y fuentes de datos estandarizadas. Cuenta con un sistema de enrutamiento agnóstico al proveedor para orquestar tareas a través de múltiples APIs de nube propietarias y backends de IA locales. El sistema incluye capacidades para la automatización de navegadores headless, análisis de contenido visual y análisis de código basado en símbolos para mapear bases de código. Para garantizar la seguridad, implementa barandillas (guardrails) con intervención humana que requieren confirmación del usuario antes de ejecutar cambios sensibles en el sistema o finalizar parches de archivos. La aplicación puede desplegarse como un binario de escritorio independiente o mediante contenedorización Docker.