Awesome GitHub RepositoriesModel Compilation Optimizers

Tinygrad is a deep learning framework and tensor computation engine designed for building and training neural networks. It functions as a hardware abstraction layer that manages device memory, command queues, and kernel dispatching across heterogeneous computing architectures. By utilizing a lazy-evaluation approach, the framework constructs computational graphs that defer execution until data is explicitly required, allowing it to process only the necessary operations for a given result. The project distinguishes itself through a just-in-time compilation layer that transforms abstract comput

This project is an educational platform and research toolkit designed to teach deep learning through a combination of mathematical theory, visual diagrams, and executable code. It provides a comprehensive environment for building, training, and evaluating neural networks, grounding complex concepts in interactive computational notebooks that allow for hands-on experimentation. The framework distinguishes itself by interleaving theoretical foundations—including linear algebra, calculus, and probability—with practical implementations across multiple industry-standard libraries. It supports flex

This project is a collection of optimized scripts, deployment patterns, and reference implementations designed for scaling and accelerating state-of-the-art AI models. It serves as a multi-domain model zoo and a distributed training framework, providing PyTorch reference implementations for training and deploying models on GPU-accelerated infrastructure. The repository distinguishes itself through an optimization suite focused on NVIDIA GPU hardware, utilizing automatic mixed precision and specialized math modes to increase training speed and throughput. It provides enterprise deployment patt

MNN is a high-performance inference engine and framework designed for on-device machine learning. It provides a comprehensive environment for executing, optimizing, and deploying neural network models directly on mobile and resource-constrained edge devices. The framework distinguishes itself through a robust model optimization toolkit that supports quantization, compression, and structural graph manipulation to minimize memory footprint and maximize execution speed. It features a modular architecture that abstracts hardware-specific backends, allowing models to run efficiently across diverse

This repository is a collection of Jupyter notebooks providing reference implementations and templates for building, training, and deploying machine learning models using Amazon SageMaker. It serves as an example library for implementing model architectures and automating the machine learning lifecycle. The library provides practical patterns for machine learning training, data engineering, and model deployment. It includes implementation guides for MLOps, including workflows for model monitoring, lineage tracking, and hyperparameter tuning. The examples cover a broad range of capabilities i

OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and

AutoGluon is an automated machine learning framework and multimodal library designed to automate the end-to-end pipeline from data preprocessing to high-accuracy model training and validation. It functions as an automated model trainer for tabular, image, text, and time series data, as well as a tool for time series forecasting and foundation model finetuning. The project is distinguished by its ability to jointly process and fuse different data types, allowing for the construction of multimodal neural networks that integrate images, text, and structured tables. It supports zero-shot inferenc

SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi

LiteRT-LM ist ein Hochleistungs-Inferenz-Framework, das darauf ausgelegt ist, Large Language Models lokal auf Mobil-, Desktop- und IoT-Hardware auszuführen. Es dient als On-Device-Modell-Laufzeitumgebung, die CPU-, GPU- und NPU-Beschleunigung nutzt, um eine Verarbeitung mit geringer Latenz zu ermöglichen. Das Framework zeichnet sich durch die Fähigkeit aus, Text-, Bild- und Audioeingaben über eine einzige multimodale Inferenz-Engine zu verarbeiten. Es verfügt über einen lokalen HTTP-Server, der OpenAI-kompatible API-Endpunkte emuliert, sowie eine WebGPU-basierte Laufzeitumgebung zur Ausführung von Modellen direkt im Webbrowser. Um die Zuverlässigkeit der Ausgabe zu gewährleisten, enthält es einen eingeschränkten Textgenerator, der JSON-Schemas oder Grammatikregeln für Modellantworten erzwingt. Das Projekt bietet umfassende Funktionen für zustandsbehaftetes Konversationsmanagement, spekulative Dekodierung für höhere Token-Generierungsgeschwindigkeiten und eine Tool-Calling-Schnittstelle, die Modellanfragen auf externe Funktionen abbildet. Es beinhaltet zudem eine spezialisierte Integration für das Apple-Ökosystem und ein dediziertes Plugin für die Modellausführung in Flutter. Benutzer können Modelle über eine Befehlszeilenschnittstelle ausführen oder sie über native APIs in Anwendungen integrieren.

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

AI-System ist eine Bildungsressource und ein Toolkit, das für das Erlernen der Hardware- und Software-Grundlagen von Deep-Learning-Systemen konzipiert ist. Es bietet einen Lehrplan und praktische Übungen zum Aufbau von KI-Infrastruktur, die von Low-Level-CUDA-Kernel-Entwicklung bis hin zu High-Level-Systemmanagement reichen. Das Projekt enthält ein Toolkit zur Entwicklung von Tensor-Operationen und zur Optimierung der GPU-Performance durch direkte Hardwareprogrammierung. Zudem bietet es ein Framework für verteiltes Training, das sich auf Ressourcenplanung und Kommunikationsprotokolle konzentriert, um groß angelegte Modelle über mehrere Rechenknoten hinweg zu verwalten. Das System deckt KI-Sicherheitsanalysen zur Identifizierung von Datenschutzschwachstellen und gegnerischen Angriffen ab sowie Performance-Optimierung durch hardwarebewusste Kompilierung, Sparsity-getriebene Kompression und tensorbasierte Berechnungsgraphen. Es bietet zudem Tools zur Verwaltung von KI-Infrastruktur und zur Koordinierung von Bereitstellungsstrategien für High-Performance-Inferenzumgebungen.

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

IREE is an MLIR-based compiler toolchain and runtime designed to translate machine learning models from various frameworks into optimized binaries for execution across diverse hardware targets. It provides a unified pipeline to ingest models from PyTorch, TensorFlow, JAX, and ONNX, lowering them into a common intermediate representation for deployment on CPUs, GPUs, and bare-metal embedded systems. The project distinguishes itself through a bytecode virtual machine and a hardware abstraction layer that decouple high-level model logic from specific hardware instruction sets. It supports sophis

FastVideo is a comprehensive system for accelerated video generation, serving as a video generation inference engine, a video diffusion training framework, and a modular pipeline orchestrator. It provides a distributed transformer optimizer and a distillation toolkit designed to reduce denoising steps and model complexity to increase frame rates. The project distinguishes itself through specialized acceleration techniques, including joint distillation and sparse attention training. It implements low-step video generation and weight quantization to FP8 or FP4 precision to increase throughput a

Neural Compressor is a deep learning model compression toolkit and AI inference acceleration engine. It functions as an automated model quantization tool and hardware-aware model compiler designed to reduce the memory footprint of neural networks and decrease execution latency. The project provides specialized frameworks for optimizing large language models, utilizing weight-only quantization and hardware-specific kernels to improve the operational efficiency of generative AI workloads. It maps neural network operators to specialized CPU and GPU vector instructions to accelerate model executi

LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT hardware. It consists of an inference engine and a specialized environment for running quantized large language and diffusion models locally on edge hardware. The system includes an ahead-of-time model compiler that translates models into hardware-specific bytecode to reduce startup latency and memory overhead. It provides a unified interface for Neural Processing Units with automatic fallback routing to CPUs or GPUs when specific subgraph support is unavailable. An edge model conve