Awesome GitHub RepositoriesModel Serving

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

This library provides a framework for parameter-efficient fine-tuning, enabling the adaptation of large pretrained models by training only a small subset of parameters. It functions as a distributed model training system and optimization toolkit, designed to reduce the computational and memory requirements typically associated with full model fine-tuning. The project distinguishes itself through a suite of methods for modular adapter composition, including low-rank matrix decomposition and activation-based scaling. It supports the integration of multiple task-specific adapter modules, allowin

The simplest way to run LLaMA on your local machine

This project is a structured learning curriculum and technical reference for mastering deep learning with TensorFlow. It provides a comprehensive guide for building, training, and deploying neural networks, combining theoretical fundamentals with practical implementation examples. The repository distinguishes itself by covering the end-to-end machine learning workflow, from low-level tensor mathematics and linear algebra to the creation of complex model architectures. It includes specific guidance on developing data pipelines for diverse data types, such as images, text, and time-series seque

Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

One Small Step is an educational resource that explains core AI and large language model concepts through short, accessible articles designed to be read in under five minutes. It covers the structure and function of key LLM components like attention mechanisms and tokenization, as well as foundational machine learning mathematics such as matrix rank and overfitting. The project also serves as a guide to the GGUF file format, which packages all model parameters and metadata into a single compact binary file for cross-platform deployment without external dependencies. It explains how this forma

StoryDiffusion is a generative AI system designed for consistent character image and video generation. It utilizes a pluggable cross-attention module to inject shared character representations into pretrained diffusion models, allowing for visual identity stability across multiple images and scenes without retraining the base model. The project features a video generation pipeline that produces temporally coherent sequences from text prompts or condition images. It employs a latent space motion interpolator to predict intermediate frames and semantic motion, enabling long-range video generati

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

FATE is an open-source federated learning platform that enables multiple organizations to collaboratively train machine learning models without exposing raw data to any party. It provides a complete framework for private data collaboration, allowing participants to jointly compute on sensitive information while maintaining data privacy and security guarantees through secure multi-party computation protocols. The platform distinguishes itself through its comprehensive infrastructure management capabilities, supporting automated deployment of multi-party clusters using Ansible-driven provisioni

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

OpenChat es un framework para el entrenamiento, ajuste fino (fine-tuning) y despliegue de modelos de lenguaje de gran tamaño optimizados para tareas de razonamiento conversacional y matemático. Proporciona un ciclo de vida completo para estos modelos, desde pipelines de entrenamiento y stacks de despliegue hasta una interfaz de chat basada en web. El proyecto se centra en permitir la ejecución de modelos de alto rendimiento en hardware de consumo sin necesidad de aceleradores de nivel empresarial. Incluye un servidor de inferencia listo para producción que implementa el protocolo de chat completion de OpenAI y utiliza el procesamiento por lotes dinámico (dynamic batching) para optimizar el rendimiento del hardware. El sistema cubre todo el flujo de trabajo operativo, incluyendo la tokenización de datasets y el ajuste fino de modelos mediante entrenamiento sin padding y aprendizaje por refuerzo. Se extiende además al alojamiento de API con autenticación basada en claves y una interfaz gráfica de usuario para la interacción humana en tiempo real.

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

Aibrix es un orquestador de inferencia diseñado para escalar, enrutar y gestionar el despliegue de modelos de lenguaje grandes a través de clústeres vLLM distribuidos. Sirve como una puerta de enlace centralizada para el balanceo de carga y el enrutamiento de tráfico a réplicas y versiones específicas de modelos. El sistema gestiona la eficiencia de los recursos a través de un autoescalador de clúster de GPU que ajusta el conteo de instancias de cómputo según el volumen de solicitudes en tiempo real. Además, optimiza las operaciones mezclando diferentes tipos de aceleradores dentro de un solo clúster y utilizando un orquestador de adaptadores de modelos para desplegar adaptadores de parámetros ligeros en modelos base compartidos. Las capacidades generales incluyen el uso de un gestor de caché de clave-valor distribuido para compartir datos de tokens a través de motores de inferencia y la implementación de monitoreo de salud del hardware para detectar fallos en las unidades de procesamiento. El proyecto también proporciona un pipeline de métricas unificado para estandarizar la recopilación de datos de rendimiento en diversos entornos de ejecución.

Este proyecto es un recurso educativo integral y un plan de estudios centrado en el diseño e implementación de todo el stack de software y hardware de aprendizaje automático. Sirve como referencia técnica para la arquitectura de sistemas de aprendizaje automático, abarcando desde interfaces de programación de bajo nivel hasta infraestructura de despliegue a gran escala. El proyecto proporciona orientación instructiva sobre varios dominios especializados, incluyendo el desarrollo de compiladores de IA a través de representaciones intermedias y optimizaciones de grafos. Cubre los patrones arquitectónicos necesarios para el entrenamiento distribuido a través de clústeres de GPU y la programación de aceleradores de hardware para optimizar cargas de trabajo en chips especializados. El recurso también detalla la implementación de frameworks de servicio de modelos para entornos de producción y el diseño de pipelines de aprendizaje por refuerzo. Su alcance se extiende a los componentes centrales de los sistemas de ML, como la diferenciación automática, abstracciones de tensores y la orquestación de recursos de GPU.