Awesome GitHub RepositoriesModel Serving

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

This library provides a framework for parameter-efficient fine-tuning, enabling the adaptation of large pretrained models by training only a small subset of parameters. It functions as a distributed model training system and optimization toolkit, designed to reduce the computational and memory requirements typically associated with full model fine-tuning. The project distinguishes itself through a suite of methods for modular adapter composition, including low-rank matrix decomposition and activation-based scaling. It supports the integration of multiple task-specific adapter modules, allowin

The simplest way to run LLaMA on your local machine

This project is a structured learning curriculum and technical reference for mastering deep learning with TensorFlow. It provides a comprehensive guide for building, training, and deploying neural networks, combining theoretical fundamentals with practical implementation examples. The repository distinguishes itself by covering the end-to-end machine learning workflow, from low-level tensor mathematics and linear algebra to the creation of complex model architectures. It includes specific guidance on developing data pipelines for diverse data types, such as images, text, and time-series seque

Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

One Small Step is an educational resource that explains core AI and large language model concepts through short, accessible articles designed to be read in under five minutes. It covers the structure and function of key LLM components like attention mechanisms and tokenization, as well as foundational machine learning mathematics such as matrix rank and overfitting. The project also serves as a guide to the GGUF file format, which packages all model parameters and metadata into a single compact binary file for cross-platform deployment without external dependencies. It explains how this forma

StoryDiffusion is a generative AI system designed for consistent character image and video generation. It utilizes a pluggable cross-attention module to inject shared character representations into pretrained diffusion models, allowing for visual identity stability across multiple images and scenes without retraining the base model. The project features a video generation pipeline that produces temporally coherent sequences from text prompts or condition images. It employs a latent space motion interpolator to predict intermediate frames and semantic motion, enabling long-range video generati

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

FATE is an open-source federated learning platform that enables multiple organizations to collaboratively train machine learning models without exposing raw data to any party. It provides a complete framework for private data collaboration, allowing participants to jointly compute on sensitive information while maintaining data privacy and security guarantees through secure multi-party computation protocols. The platform distinguishes itself through its comprehensive infrastructure management capabilities, supporting automated deployment of multi-party clusters using Ansible-driven provisioni

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

OpenChat est un framework pour l'entraînement, le fine-tuning et le déploiement de grands modèles de langage optimisés pour les tâches de raisonnement conversationnel et mathématique. Il fournit un cycle de vie complet pour ces modèles, allant des pipelines d'entraînement et des stacks de déploiement à une interface de chat web. Le projet se concentre sur l'activation d'une exécution de modèle haute performance sur du matériel grand public sans avoir besoin d'accélérateurs de classe entreprise. Il inclut un serveur d'inférence prêt pour la production qui implémente le protocole de complétion de chat OpenAI et utilise le batching dynamique des requêtes pour optimiser le débit matériel. Le système couvre l'ensemble du flux de travail opérationnel, y compris la tokenisation des jeux de données et le fine-tuning des modèles via un entraînement sans padding et l'apprentissage par renforcement. Il s'étend également à l'hébergement d'API avec authentification par clé et une interface graphique pour l'interaction humaine en temps réel.

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

Aibrix est un orchestrateur d'inférence conçu pour la mise à l'échelle, le routage et la gestion du déploiement de modèles de langage étendus à travers des clusters vLLM distribués. Il sert de passerelle centralisée pour l'équilibrage de charge et le routage du trafic vers des répliques et des versions de modèles spécifiques. Le système gère l'efficacité des ressources via un autoscaler de cluster GPU qui ajuste le nombre d'instances de calcul en fonction du volume de requêtes en temps réel. Il optimise davantage les opérations en mélangeant différents types d'accélérateurs au sein d'un même cluster et en utilisant un orchestrateur d'adaptateurs de modèles pour déployer des adaptateurs de paramètres légers sur des modèles de base partagés. Les capacités étendues incluent l'utilisation d'un gestionnaire de cache clé-valeur distribué pour partager les données de jetons à travers les moteurs d'inférence et l'implémentation de la surveillance de la santé du matériel pour détecter les défaillances des unités de traitement. Le projet fournit également un pipeline de métriques unifié pour standardiser la collecte de données de performance à travers divers environnements d'exécution.

Ce projet est une ressource éducative complète et un programme axé sur la conception et l'implémentation de la pile logicielle et matérielle complète du machine learning. Il sert de référence technique pour l'architecture des systèmes de machine learning, allant des interfaces de programmation de bas niveau à l'infrastructure de déploiement à grande échelle. Le projet fournit des conseils pédagogiques sur plusieurs domaines spécialisés, notamment le développement de compilateurs IA via des représentations intermédiaires et des optimisations de graphes. Il couvre les modèles architecturaux requis pour l'entraînement distribué sur des clusters GPU et la programmation d'accélérateurs matériels pour optimiser les charges de travail sur des puces spécialisées. La ressource détaille également l'implémentation de frameworks de service de modèles pour les environnements de production et la conception de pipelines d'apprentissage par renforcement. Sa portée s'étend aux composants de base des systèmes ML, tels que la différenciation automatique, les abstractions de tenseurs et l'orchestration des ressources GPU.