Awesome GitHub RepositoriesInference Optimization Kernels

BitNet is a quantized inference engine designed to execute highly compressed language models by performing arithmetic on low-precision, bit-level weight data. It functions as a model optimization toolkit and a high-performance kernel library, enabling the execution of large language models on consumer hardware by reducing memory footprints and increasing processing speeds. The project distinguishes itself through hardware-specific kernel optimizations that leverage native processor instructions to accelerate matrix multiplication. By utilizing packed integer arithmetic and memory-aligned weig

This repository serves as a comprehensive collection of reference implementations for the PyTorch machine learning library. It provides practical examples for building, training, and deploying deep learning models, functioning as a toolkit for developers to explore neural network architectures and training workflows. The project distinguishes itself by offering concrete demonstrations of complex machine learning operations, ranging from computer vision tasks like object detection and depth estimation to the training of large-scale transformer models. These examples illustrate how to implement

This project is a comprehensive framework for the training, fine-tuning, and deployment of large language models. It functions as a distributed deep learning platform that enables users to scale model workflows across multiple hardware nodes while providing tools for model evaluation and performance benchmarking. The platform distinguishes itself by offering specialized utilities for model compression and weight transformation, allowing users to reduce memory footprints and latency through quantization and pruning. It supports the adaptation of large models for consumer-grade hardware, facili

Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode

Mamba is a deep learning framework designed for building and training sequence models that process long-range data dependencies with linear-time computational efficiency. By utilizing selective state space modeling, the library enables the construction of neural network architectures that replace traditional attention mechanisms with high-performance state space operations. The framework distinguishes itself through the use of data-dependent state gating, which allows the model to dynamically filter information flow based on the input sequence. To ensure high throughput, it incorporates hardw

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com

lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models. It functions as a multi-modal model server and compression toolkit designed to serve models with high throughput and low latency. The system enables the distribution of model services across multiple machines using request-based load balancing and tensor parallelism. It includes specialized tools for model quantization and compression to reduce the memory footprint of weights and caches. The framework covers broad capability areas including production deployment, distributed

Cactus es un motor de inferencia de IA en el dispositivo diseñado para ejecutar modelos de lenguaje de gran tamaño (LLM), modelos de visión y sistemas de voz a texto en hardware móvil y wearable. Proporciona un grafo de computación tensorial programable para definir secuencias de operaciones matriciales y funciones de activación, junto con un framework de RAG (generación aumentada por recuperación) local que fundamenta las respuestas del modelo utilizando archivos de texto locales. El proyecto cuenta con un SDK multiplataforma con bindings de lenguaje para integrar capacidades de IA en aplicaciones móviles y un sistema de conversión de modelos que transforma formatos externos para una ejecución local optimizada. Utiliza un sistema de enrutamiento híbrido para redirigir cargas de trabajo entre la ejecución en el dispositivo y proveedores en la nube según la capacidad del hardware. El motor cubre una amplia superficie de capacidades, incluyendo procesamiento de audio en el dispositivo para detección de actividad de voz y transcripción, generación de embeddings vectoriales para búsqueda por similitud e integración de herramientas para analizar salidas del modelo en llamadas a funciones externas. Estos procesos están respaldados por kernels nativos optimizados para un rendimiento de baja latencia en hardware móvil.

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

AutoGPTQ es un kit de herramientas de compresión de modelos y un framework de cuantización post-entrenamiento diseñado para reducir la huella de memoria de modelos de lenguaje grandes. Utiliza el algoritmo GPTQ para comprimir los pesos de las redes neuronales, reduciendo los requisitos de hardware y el uso de VRAM. El proyecto sirve como un acelerador de inferencia al proporcionar kernels optimizados que aumentan la velocidad de generación de tokens. Cuenta con extensibilidad de arquitectura de modelo, permitiendo que las capacidades de cuantización se añadan a nuevas estructuras de modelos mediante patrones configurables. El framework cubre una tubería de cuantización integral, incluyendo compresión de pesos por capa, estimación de escala basada en calibración y mapeo de memoria específico de precisión. También incluye sistemas para la evaluación del rendimiento del modelo para medir el impacto de la cuantización en la precisión en tareas de lenguaje y resumen.

tiny-llm is a large language model inference engine and transformer model implementation. It serves as a quantized model runtime and paged key-value cache manager, providing a specialized inference stack optimized for Apple Silicon. The system distinguishes itself through high-throughput execution techniques, including continuous batching and paged attention. It utilizes a paged memory system to eliminate fragmentation during token generation and employs on-the-fly dequantization of compressed weights to reduce the memory footprint during matrix multiplication. The project covers a broad ran

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b