Awesome GitHub RepositoriesDistributed Model Execution

Llama is a large language model runtime and inference engine designed to load and execute autoregressive transformer models. It enables the generation of natural language text completions from prompts using pretrained weights. The system features multi-GPU model parallelism, which distributes model weights and workloads across multiple graphics processors to support larger parameter counts. It also incorporates a content safety filter that uses classifiers to intercept and block unsafe inputs or outputs during the inference process. The project covers broad capabilities in distributed model

Grok-1 is an open-weights large language model implementation featuring a sparse mixture-of-experts architecture. It is designed for high-performance text generation and natural language processing by activating only a subset of specialized expert layers per token. The model utilizes 8-bit weight quantization to reduce memory overhead and accelerate loading. To manage its high parameter count, the implementation supports activation sharding, which distributes the memory load across multiple hardware devices during execution. The project covers large-scale model inference, including text comp

Gensim is an unsupervised natural language processing toolkit designed for topic modeling, word embedding training, and the processing of large-scale text corpora. It provides a framework for discovering latent themes and semantic structures in text without the need for labeled data. The toolkit is distinguished by its ability to handle datasets that exceed system memory through iterator-based data streaming from disk. It also supports distributed model training, allowing complex modeling tasks to be executed across computer clusters. The library covers a broad range of analysis capabilities

ds4 is a local inference engine for DeepSeek models that includes a distributed runtime for splitting transformer layers across networked computers. It functions as a reasoning controller with a local weight streamer and an API server that streams chat completions via industry standard endpoints. The system employs a memory management model that loads model experts from disk on demand to execute models that exceed available system RAM. It provides controls for reasoning effort and model behavior steering, allowing the modification of response characteristics through activation directions. Th

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

Este proyecto es una biblioteca y framework de inferencia de modelos de lenguaje de gran tamaño (LLM) diseñado para ejecutar modelos para generación de texto, resolución de problemas y asistencia en codificación. Incluye un framework multimodal para procesar entradas combinadas de imagen y texto, y una implementación de uso de herramientas que permite la ejecución de funciones externas basadas en el razonamiento del modelo. El sistema cuenta con un motor de inferencia de GPU distribuido que reparte las cargas de trabajo de modelos grandes a través de múltiples procesadores gráficos para aumentar la velocidad de procesamiento y cumplir con los requisitos de memoria. También proporciona despliegue de modelos en contenedores a través de imágenes preempaquetadas y dependencias para servir motores de inferencia en entornos aislados. La biblioteca cubre una gama de capacidades que incluyen análisis de entrada multimodal, integración de llamadas a funciones y codificación de relleno (fill-in-the-middle) para predecir segmentos de código faltantes. Además, admite chat interactivo con el modelo a través de una interfaz de línea de comandos para mantener sesiones conversacionales.

OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and

This project is a structured learning curriculum and technical reference for mastering deep learning with TensorFlow. It provides a comprehensive guide for building, training, and deploying neural networks, combining theoretical fundamentals with practical implementation examples. The repository distinguishes itself by covering the end-to-end machine learning workflow, from low-level tensor mathematics and linear algebra to the creation of complex model architectures. It includes specific guidance on developing data pipelines for diverse data types, such as images, text, and time-series seque

OneFlow is a deep learning framework and distributed execution engine designed for building, training, and deploying neural network architectures. It functions as a scalable neural network library that allows for the development of deep learning models and their execution across distributed hardware. The project includes a machine learning graph compiler used to optimize neural network execution graphs. This allows for the acceleration of model performance and the reduction of latency during both training and inference. The framework covers broad capability areas including large-scale model

BigDL es un framework de aceleración de PyTorch y motor de inferencia distribuida diseñado para grandes modelos de lenguaje. Proporciona un kit de herramientas para ejecutar modelos en hardware Intel, integrando herramientas de cuantización y librerías para el ajuste fino eficiente en parámetros. El proyecto se distingue por el uso de paralelismo de pipeline para distribuir cargas de trabajo de modelos a través de múltiples aceleradores de hardware. Utiliza cuantización de enteros de bajo bit y decodificación especulativa para reducir la huella de memoria y disminuir la latencia de generación de texto. El sistema cubre amplias capacidades en optimización de modelos, incluyendo compresión de pesos y carga de modelos cuantizados. También admite rutinas de entrenamiento aceleradas por hardware para adaptar modelos preentrenados a tareas específicas.

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte

BentoML is a machine learning model serving framework and GPU-accelerated inference server designed to package, deploy, and scale AI models as production-ready REST APIs. It functions as an AI model lifecycle manager and an inference graph orchestrator, enabling the chaining of multiple models and custom logic into complex pipelines for advanced task sequences. The framework distinguishes itself through a dynamic batching engine that optimizes GPU throughput and an artifact-based packaging system that bundles model weights and dependencies into immutable archives for consistent deployment. It

Caffe2 is a high-performance deep learning framework and C++ machine learning library. It serves as a modular system for designing, training, and executing scalable neural networks. The project functions as an inference engine and a scalable neural network engine designed to run models across distributed systems and diverse hardware. Its architecture allows for the construction of custom neural network components that can be scaled from research to production environments. The framework covers the full lifecycle of deep learning development, including modular network architecture design, mod

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

Obliteratus is a weight ablation framework and refusal removal tool designed to identify and delete the internal representations responsible for content refusals in large language models without retraining. It functions as a circuit analysis suite that maps the geometric structure of model guardrails to isolate the specific layers and attention heads that enforce refusals. The project enables the removal of these behaviors through geometric projection, rank-1 adapter ablation for reversible modifications, and the application of steering vectors to alter behavior during inference. It includes

IREE is an MLIR-based compiler toolchain and runtime designed to translate machine learning models from various frameworks into optimized binaries for execution across diverse hardware targets. It provides a unified pipeline to ingest models from PyTorch, TensorFlow, JAX, and ONNX, lowering them into a common intermediate representation for deployment on CPUs, GPUs, and bare-metal embedded systems. The project distinguishes itself through a bytecode virtual machine and a hardware abstraction layer that decouple high-level model logic from specific hardware instruction sets. It supports sophis

This is a TensorFlow-based framework for implementing and training capsule networks designed for image classification. It utilizes a tensor graph neural network model to build architectures that preserve spatial relationships and pose parameters through vector-based neuron groupings. The project implements a dynamic routing mechanism to establish hierarchical relationships between input features and output capsules. To handle complex architectures, it includes a distributed deep learning toolset that enables training across multiple graphics processing units. The framework covers the full ma

Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute manager designed to deploy and scale model workloads across diverse hardware architectures, including GPUs, CPUs, and heterogeneous hardware clusters. The platform enables model deployment across a wide range of targets, including NVIDIA GPUs, regional chipsets, and legacy hardware. It manages the execution of models across these varying environments to increase available computing capacity and optimize resource utilization. The system includes capabilities for distributed i

zml is a machine learning model compiler and cross-platform inference engine that transforms model descriptions into optimized executable binaries for specific hardware accelerators. It functions as a model deployment toolkit and hardware-agnostic orchestrator, utilizing a tensor-based architecture definition to provide strong type checking during the compilation process. The project distinguishes itself through the ability to shard tensors and distribute large-scale AI workloads across a logical mesh of multiple devices. It further supports the remote model lifecycle by authenticating and do