Awesome GitHub RepositoriesModel Serving

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

This library provides a framework for parameter-efficient fine-tuning, enabling the adaptation of large pretrained models by training only a small subset of parameters. It functions as a distributed model training system and optimization toolkit, designed to reduce the computational and memory requirements typically associated with full model fine-tuning. The project distinguishes itself through a suite of methods for modular adapter composition, including low-rank matrix decomposition and activation-based scaling. It supports the integration of multiple task-specific adapter modules, allowin

The simplest way to run LLaMA on your local machine

This project is a structured learning curriculum and technical reference for mastering deep learning with TensorFlow. It provides a comprehensive guide for building, training, and deploying neural networks, combining theoretical fundamentals with practical implementation examples. The repository distinguishes itself by covering the end-to-end machine learning workflow, from low-level tensor mathematics and linear algebra to the creation of complex model architectures. It includes specific guidance on developing data pipelines for diverse data types, such as images, text, and time-series seque

Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

One Small Step is an educational resource that explains core AI and large language model concepts through short, accessible articles designed to be read in under five minutes. It covers the structure and function of key LLM components like attention mechanisms and tokenization, as well as foundational machine learning mathematics such as matrix rank and overfitting. The project also serves as a guide to the GGUF file format, which packages all model parameters and metadata into a single compact binary file for cross-platform deployment without external dependencies. It explains how this forma

StoryDiffusion is a generative AI system designed for consistent character image and video generation. It utilizes a pluggable cross-attention module to inject shared character representations into pretrained diffusion models, allowing for visual identity stability across multiple images and scenes without retraining the base model. The project features a video generation pipeline that produces temporally coherent sequences from text prompts or condition images. It employs a latent space motion interpolator to predict intermediate frames and semantic motion, enabling long-range video generati

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

FATE is an open-source federated learning platform that enables multiple organizations to collaboratively train machine learning models without exposing raw data to any party. It provides a complete framework for private data collaboration, allowing participants to jointly compute on sensitive information while maintaining data privacy and security guarantees through secure multi-party computation protocols. The platform distinguishes itself through its comprehensive infrastructure management capabilities, supporting automated deployment of multi-party clusters using Ansible-driven provisioni

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

OpenChat संवादात्मक और गणितीय तर्क कार्यों के लिए अनुकूलित लार्ज लैंग्वेज मॉडल के प्रशिक्षण, फाइन-ट्यूनिंग और परिनियोजन (deployment) के लिए एक फ्रेमवर्क है। यह इन मॉडलों के लिए एक व्यापक लाइफसाइकिल प्रदान करता है, जिसमें ट्रेनिंग पाइपलाइन और डिप्लॉयमेंट स्टैक से लेकर वेब-आधारित चैट इंटरफ़ेस तक शामिल है। यह प्रोजेक्ट एंटरप्राइज़-ग्रेड एक्सेलेरेटर की आवश्यकता के बिना उपभोक्ता-ग्रेड हार्डवेयर पर उच्च-प्रदर्शन मॉडल निष्पादन को सक्षम करने पर केंद्रित है। इसमें एक प्रोडक्शन-रेडी इन्फरेंस सर्वर शामिल है जो OpenAI चैट कंप्लीशन प्रोटोकॉल को लागू करता है और हार्डवेयर थ्रूपुट को ऑप्टिमाइज़ करने के लिए डायनामिक रिक्वेस्ट बैचिंग का उपयोग करता है। यह सिस्टम संपूर्ण परिचालन वर्कफ़्लो को कवर करता है, जिसमें पैडिंग-फ्री ट्रेनिंग और रीइन्फोर्समेंट लर्निंग के माध्यम से डेटासेट टोकनाइज़ेशन और मॉडल फाइन-ट्यूनिंग शामिल है। यह की-आधारित प्रमाणीकरण के साथ API होस्टिंग और वास्तविक समय मानवीय बातचीत के लिए एक ग्राफिकल यूजर इंटरफेस तक विस्तारित है।

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

Aibrix वितरित vLLM क्लस्टर में लार्ज लैंग्वेज मॉडल के डिप्लॉयमेंट को स्केल करने, रूट करने और प्रबंधित करने के लिए डिज़ाइन किया गया एक इन्फरेंस ऑर्केस्ट्रेटर है। यह लोड बैलेंसिंग और विशिष्ट मॉडल प्रतिकृतियों (replicas) व संस्करणों के लिए ट्रैफिक रूटिंग के लिए एक केंद्रीकृत गेटवे के रूप में कार्य करता है। यह सिस्टम GPU क्लस्टर ऑटोस्केलर के माध्यम से संसाधन दक्षता का प्रबंधन करता है जो रीयल-टाइम रिक्वेस्ट वॉल्यूम के आधार पर कंप्यूट इंस्टेंस काउंट को समायोजित करता है। यह एक एकल क्लस्टर के भीतर विभिन्न एक्सेलेरेटर प्रकारों को मिलाकर और साझा बेस मॉडल पर लाइटवेट पैरामीटर एडेप्टर डिप्लॉय करने के लिए एक मॉडल एडेप्टर ऑर्केस्ट्रेटर का उपयोग करके संचालन को और अनुकूलित करता है। व्यापक क्षमताओं में इन्फरेंस इंजन में टोकन डेटा साझा करने के लिए एक वितरित की-वैल्यू कैश मैनेजर का उपयोग और प्रोसेसिंग यूनिट विफलताओं का पता लगाने के लिए हार्डवेयर स्वास्थ्य निगरानी का कार्यान्वयन शामिल है। यह प्रोजेक्ट विविध रनटाइम वातावरणों में प्रदर्शन डेटा संग्रह को मानकीकृत करने के लिए एक एकीकृत मेट्रिक्स पाइपलाइन भी प्रदान करता है।

This project is a comprehensive educational resource and curriculum focused on the design and implementation of the full machine learning software and hardware stack. It serves as a technical reference for architecting machine learning systems, spanning from low-level programming interfaces to large-scale deployment infrastructure. The project provides instructional guidance on several specialized domains, including the development of AI compilers through intermediate representations and graph optimizations. It covers the architectural patterns required for distributed training across GPU clu