Awesome GitHub RepositoriesDistributed Model Execution

Llama is a large language model runtime and inference engine designed to load and execute autoregressive transformer models. It enables the generation of natural language text completions from prompts using pretrained weights. The system features multi-GPU model parallelism, which distributes model weights and workloads across multiple graphics processors to support larger parameter counts. It also incorporates a content safety filter that uses classifiers to intercept and block unsafe inputs or outputs during the inference process. The project covers broad capabilities in distributed model

Grok-1 is an open-weights large language model implementation featuring a sparse mixture-of-experts architecture. It is designed for high-performance text generation and natural language processing by activating only a subset of specialized expert layers per token. The model utilizes 8-bit weight quantization to reduce memory overhead and accelerate loading. To manage its high parameter count, the implementation supports activation sharding, which distributes the memory load across multiple hardware devices during execution. The project covers large-scale model inference, including text comp

Gensim is an unsupervised natural language processing toolkit designed for topic modeling, word embedding training, and the processing of large-scale text corpora. It provides a framework for discovering latent themes and semantic structures in text without the need for labeled data. The toolkit is distinguished by its ability to handle datasets that exceed system memory through iterator-based data streaming from disk. It also supports distributed model training, allowing complex modeling tasks to be executed across computer clusters. The library covers a broad range of analysis capabilities

ds4 is a local inference engine for DeepSeek models that includes a distributed runtime for splitting transformer layers across networked computers. It functions as a reasoning controller with a local weight streamer and an API server that streams chat completions via industry standard endpoints. The system employs a memory management model that loads model experts from disk on demand to execute models that exceed available system RAM. It provides controls for reasoning effort and model behavior steering, allowing the modification of response characteristics through activation directions. Th

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

This project is a large language model inference library and framework designed to run models for text generation, problem solving, and coding assistance. It includes a multimodal framework for processing combined image and text inputs and a tool-use implementation that enables the execution of external functions based on model reasoning. The system features a distributed GPU inference engine that spreads large model workloads across multiple graphics processors to increase processing speed and meet memory requirements. It also provides containerized model deployment through pre-packaged imag

OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and

This project is a structured learning curriculum and technical reference for mastering deep learning with TensorFlow. It provides a comprehensive guide for building, training, and deploying neural networks, combining theoretical fundamentals with practical implementation examples. The repository distinguishes itself by covering the end-to-end machine learning workflow, from low-level tensor mathematics and linear algebra to the creation of complex model architectures. It includes specific guidance on developing data pipelines for diverse data types, such as images, text, and time-series seque

OneFlow is a deep learning framework and distributed execution engine designed for building, training, and deploying neural network architectures. It functions as a scalable neural network library that allows for the development of deep learning models and their execution across distributed hardware. The project includes a machine learning graph compiler used to optimize neural network execution graphs. This allows for the acceleration of model performance and the reduction of latency during both training and inference. The framework covers broad capability areas including large-scale model

BigDL एक PyTorch एक्सेलेरेशन फ्रेमवर्क और डिस्ट्रीब्यूटेड इन्फरेंस इंजन है जिसे लार्ज लैंग्वेज मॉडल्स के लिए डिज़ाइन किया गया है। यह Intel हार्डवेयर पर मॉडल्स चलाने के लिए एक टूलकिट प्रदान करता है, जिसमें क्वांटाइज़ेशन टूल्स और पैरामीटर-कुशल फाइन-ट्यूनिंग के लिए लाइब्रेरीज़ शामिल हैं। यह प्रोजेक्ट कई हार्डवेयर एक्सेलेरेटर्स में मॉडल वर्कलोड को वितरित करने के लिए पाइपलाइन पैरेललिज़्म के उपयोग के माध्यम से खुद को अलग बनाता है। यह मेमोरी फुटप्रिंट को कम करने और टेक्स्ट जनरेशन लेटेंसी को घटाने के लिए लो-बिट इंटीजर क्वांटाइज़ेशन और सट्टा डिकोडिंग (speculative decoding) का उपयोग करता है। यह सिस्टम मॉडल ऑप्टिमाइज़ेशन में व्यापक क्षमताओं को कवर करता है, जिसमें वेट कम्प्रेशन और क्वांटाइज़्ड मॉडल लोडिंग शामिल है। यह प्री-ट्रेंड मॉडल्स को विशिष्ट कार्यों के अनुकूल बनाने के लिए हार्डवेयर-एक्सेलेरेटेड ट्रेनिंग रूटीन को भी सपोर्ट करता है।

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte

BentoML is a machine learning model serving framework and GPU-accelerated inference server designed to package, deploy, and scale AI models as production-ready REST APIs. It functions as an AI model lifecycle manager and an inference graph orchestrator, enabling the chaining of multiple models and custom logic into complex pipelines for advanced task sequences. The framework distinguishes itself through a dynamic batching engine that optimizes GPU throughput and an artifact-based packaging system that bundles model weights and dependencies into immutable archives for consistent deployment. It

Caffe2 is a high-performance deep learning framework and C++ machine learning library. It serves as a modular system for designing, training, and executing scalable neural networks. The project functions as an inference engine and a scalable neural network engine designed to run models across distributed systems and diverse hardware. Its architecture allows for the construction of custom neural network components that can be scaled from research to production environments. The framework covers the full lifecycle of deep learning development, including modular network architecture design, mod

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

Obliteratus is a weight ablation framework and refusal removal tool designed to identify and delete the internal representations responsible for content refusals in large language models without retraining. It functions as a circuit analysis suite that maps the geometric structure of model guardrails to isolate the specific layers and attention heads that enforce refusals. The project enables the removal of these behaviors through geometric projection, rank-1 adapter ablation for reversible modifications, and the application of steering vectors to alter behavior during inference. It includes

IREE is an MLIR-based compiler toolchain and runtime designed to translate machine learning models from various frameworks into optimized binaries for execution across diverse hardware targets. It provides a unified pipeline to ingest models from PyTorch, TensorFlow, JAX, and ONNX, lowering them into a common intermediate representation for deployment on CPUs, GPUs, and bare-metal embedded systems. The project distinguishes itself through a bytecode virtual machine and a hardware abstraction layer that decouple high-level model logic from specific hardware instruction sets. It supports sophis

This is a TensorFlow-based framework for implementing and training capsule networks designed for image classification. It utilizes a tensor graph neural network model to build architectures that preserve spatial relationships and pose parameters through vector-based neuron groupings. The project implements a dynamic routing mechanism to establish hierarchical relationships between input features and output capsules. To handle complex architectures, it includes a distributed deep learning toolset that enables training across multiple graphics processing units. The framework covers the full ma

Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute manager designed to deploy and scale model workloads across diverse hardware architectures, including GPUs, CPUs, and heterogeneous hardware clusters. The platform enables model deployment across a wide range of targets, including NVIDIA GPUs, regional chipsets, and legacy hardware. It manages the execution of models across these varying environments to increase available computing capacity and optimize resource utilization. The system includes capabilities for distributed i

zml is a machine learning model compiler and cross-platform inference engine that transforms model descriptions into optimized executable binaries for specific hardware accelerators. It functions as a model deployment toolkit and hardware-agnostic orchestrator, utilizing a tensor-based architecture definition to provide strong type checking during the compilation process. The project distinguishes itself through the ability to shard tensors and distribute large-scale AI workloads across a logical mesh of multiple devices. It further supports the remote model lifecycle by authenticating and do