Awesome GitHub RepositoriesModel Inference Optimizations

This project is a community-driven knowledge base and curated repository focused on natural language processing and large language model development. It serves as a centralized index for high-quality tools, libraries, and research materials, organizing technical resources into structured, version-controlled documentation to assist developers in navigating the evolving artificial intelligence ecosystem. The repository distinguishes itself by acting as an aggregator for AI model evaluation and benchmarking. It provides access to tools that enable the simultaneous comparison of multiple conversa

Qwen-7B is a pretrained causal language model designed for natural language generation, text processing, and complex reasoning tasks. It is available as an instruction-tuned model optimized for conversational interactions and a tool-use model capable of executing function calls and interacting with external APIs. The project provides a quantized version of the model to reduce GPU memory usage and supports the development of autonomous agents that can execute code and perform functions to complete complex goals. The system covers a wide range of capabilities including model fine-tuning throug

gpt-oss is an open-weight large language model and reasoning engine designed for complex reasoning and agentic workflows. It functions as an AI agent framework and model serving API, allowing for local deployment and the hosting of standardized interfaces to expose model completions and internal reasoning processes. The project distinguishes itself as a quantized inference engine, utilizing tensor parallelism and weight quantization to run high-parameter models on limited hardware. It features a reasoning model that employs chain-of-thought processing to solve multi-step logical tasks. The s

Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode

ChatGLM3 is a comprehensive framework for deploying, fine-tuning, and serving large language models. It functions as a high-performance inference engine designed to support conversational AI, enabling developers to build interactive agents capable of multi-turn dialogue, autonomous code execution, and structured tool invocation. The project distinguishes itself through its focus on hardware-agnostic deployment and resource optimization. It supports distributed model parallelism across multiple graphics cards, paged key-value caching for concurrent request processing, and weight quantization t

TensorRT एक डीप लर्निंग इन्फरेंस इंजन और सॉफ्टवेयर डेवलपमेंट किट है जिसे NVIDIA GPUs पर उच्च-प्रदर्शन निष्पादन के लिए न्यूरल नेटवर्क को ऑप्टिमाइज़ और डिप्लॉय करने के लिए डिज़ाइन किया गया है। यह एक GPU एक्सेलेरेशन फ्रेमवर्क के रूप में कार्य करता है जो प्रोडक्शन डिप्लॉयमेंट के दौरान प्रशिक्षित मॉडलों के लिए लेटेंसी को कम करता है और थ्रूपुट को बढ़ाता है। यह टूलकिट Open Neural Network Exchange फॉर्मेट से मॉडल इम्पोर्ट करता है और उन्हें ऑप्टिमाइज़्ड इंजनों में बदल देता है। यह ग्राफ-आधारित मॉडल ऑप्टिमाइज़ेशन, लेयर-फ्यूजन कर्नल जनरेशन, और फ्लोटिंग पॉइंट वेट्स को लोअर प्रिसिजन फॉर्मेट में बदलने के लिए प्रिसिजन-आधारित क्वांटिज़ेशन का उपयोग करता है। यह फ्रेमवर्क हार्डवेयर-विशिष्ट इंजन सीरियलाइज़ेशन के लिए क्षमताएं प्रदान करता है और विशेष न्यूरल नेटवर्क लेयर्स के लिए कस्टम प्लगइन्स के माध्यम से इन्फरेंस क्षमताओं के विस्तार का समर्थन करता है।

Stable Diffusion WebUI Forge is a web-based interface and inference engine designed for the generation of AI media. It functions as a platform for executing diffusion-based models, providing a centralized environment to manage image preprocessors, custom generation logic, and hardware-accelerated sampling. The project distinguishes itself through a neural network patching framework that allows for the modification of model layers and the application of spatial conditioning during inference. By injecting custom logic and adapters directly into the network, users can influence output behaviors

This project is a comprehensive technical reference and educational resource focused on the lifecycle of large language models. It provides structured learning materials that cover the foundational mechanics of transformer architectures, the mathematical principles of attention mechanisms, and the engineering practices required for modern generative artificial intelligence. The repository serves as a guide for both technical skill development and professional preparation, offering a curriculum that spans from model training and inference optimization to advanced alignment techniques. It detai

PowerInfer is an inference engine and serving framework designed to run large language models on local hardware. It combines a hybrid CPU-GPU offloader, a quantization tool, and a sparse model optimizer to enable the execution of high-parameter models on consumer-grade devices. The system distinguishes itself through neuron-activation-based offloading, using a predictor model to preload frequent neurons into VRAM while keeping rare neurons in system memory. This hybrid execution model balances workloads between the GPU and CPU based on input patterns to optimize memory access and increase tok

This project is an AI-powered IDE extension and LLM coding assistant that provides a conversational interface for generating, refactoring, and debugging code. It functions as an AI agent framework and a Model Context Protocol client, connecting AI models to external data sources and tools to automate complex development tasks. The system is distinguished by its use of autonomous AI agents capable of multi-step task execution, including the ability to read files, modify code, and run terminal commands iteratively. It supports recursive agent orchestration through subagent delegation and employ

MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp

OpenChatKit is a training and inference toolkit for large language models. It provides a comprehensive set of tools for managing the model lifecycle, including a fine-tuning pipeline, a model weight converter, and a command-line interface for interacting with conversational agents. The toolkit features a framework for retrieval augmented generation, allowing models to incorporate relevant context from external vector indices. It also includes utilities for converting trained model checkpoints into formats compatible with standard inference libraries. The project covers conversational AI trai

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

ipex-llm is an acceleration library and inference engine designed to optimize the execution and finetuning of large language models on Intel GPUs and NPUs. It provides a HuggingFace compatible model backend and a dedicated quantization toolkit for converting model weights into low-bit precision formats. The project facilitates distributed inference by splitting large model workloads across multiple accelerators using pipeline and tensor parallelism. It enables the deployment of models on Intel Arc, Flex, and Max GPUs to increase throughput and reduce latency. The library covers a broad range

Boxmot is a multi-object tracking framework designed to follow multiple objects across video frames using motion and appearance algorithms to maintain consistent identities. It functions as a system for tracking objects with specific orientations using rotated bounding boxes and corresponding intersection-over-union computations. The project includes a re-identification model optimizer that converts neural networks into formats for hardware-accelerated execution. It also features an evolutionary hyperparameter tuner that iteratively mutates tracker settings to maximize accuracy for specific d

pysheeet एक तकनीकी संदर्भ लाइब्रेरी है जो उन्नत Python डेवलपमेंट, सिस्टम एकीकरण और उच्च-प्रदर्शन कंप्यूटिंग के लिए कोड स्निपेट्स और कार्यान्वयन पैटर्न का एक क्यूरेटेड संग्रह प्रदान करती है। यह निम्न-स्तरीय नेटवर्क प्रोग्रामिंग, नेटिव C एक्सटेंशन, और एसिंक्रोनस और समवर्ती प्रोग्रामिंग को लागू करने के लिए एक व्यापक गाइड के रूप में कार्य करती है। यह प्रोजेक्ट वितरित GPU इन्फरेंस और उच्च-प्रदर्शन सर्विंग के लिए टूल्स सहित लार्ज लैंग्वेज मॉडल्स के विकास और डिप्लॉयमेंट के लिए विशेष फ्रेमवर्क प्रदान करती है। इसमें GPU रिसोर्स एलोकेशन और मल्टी-नोड वर्कलोड मैनेजमेंट को कवर करते हुए उच्च-प्रदर्शन कंप्यूटिंग क्लस्टर ऑर्केस्ट्रेशन के लिए विस्तृत पैटर्न भी शामिल हैं। यह लाइब्रेरी सुरक्षित नेटवर्क संचार और क्रिप्टोग्राफी, ऑब्जेक्ट-रिलेशनल मैपिंग और डेटाबेस मैनेजमेंट, और जटिल डेटा स्ट्रक्चर्स और एल्गोरिदम के कार्यान्वयन सहित क्षमताओं की एक विस्तृत श्रृंखला को कवर करती है। यह मेमोरी मैनेजमेंट, फॉरेन-फंक्शन इंटरफ़ेस के माध्यम से नेटिव इंटरऑपरेबिलिटी, और सिस्टम-स्तरीय OS एकीकरण के लिए यूटिलिटीज़ भी प्रदान करती है।

DeepSpeedExamples is a collection of reference implementations for training and deploying large scale AI models using the DeepSpeed optimization library. It provides Python code examples for training massive models across multiple GPUs through distributed optimization techniques. The repository includes optimized patterns for deploying and running large language model predictions in production environments. It also serves as a guide for model compression to reduce memory footprints and as a source for performance benchmarks to measure execution speed and resource utilization. The project cov

One Small Step is an educational resource that explains core AI and large language model concepts through short, accessible articles designed to be read in under five minutes. It covers the structure and function of key LLM components like attention mechanisms and tokenization, as well as foundational machine learning mathematics such as matrix rank and overfitting. The project also serves as a guide to the GGUF file format, which packages all model parameters and metadata into a single compact binary file for cross-platform deployment without external dependencies. It explains how this forma

I've made everythign less reliant on torch.compile for VRAM efficiency, so things should work better even without it. Also figured workaround for some issues when using compile that made first run use drastically more VRAM, issue I battled with myself a lot.

This project is a deep learning implementation of the RetinaNet architecture for detecting and classifying objects within images. Built as a Keras object detection framework and a TensorFlow computer vision tool, it provides a complete neural network implementation based on the RetinaNet paper. The framework includes specialized components such as a Feature Pyramid Network and a focal loss function to handle object detection. It features a configurable backbone architecture and anchor-based bounding boxes to predict object locations across varying scales and aspect ratios. The toolset covers