Awesome GitHub RepositoriesDistributed Gradient Synchronization

This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed

XGBoost is a distributed machine learning library for implementing scalable gradient boosting decision trees used for regression, classification, and ranking. It functions as a predictive model framework and a cross-language toolkit, providing a core implementation with native bindings for Python, R, Java, Scala, and C++. The system is designed as a GPU-accelerated library that utilizes CUDA and NCCL to speed up the training of decision tree ensembles. It operates as a distributed framework capable of scaling training and prediction across multi-node clusters and GPU environments to process m

This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en

This project is a comprehensive educational resource and technical documentation suite for learning and developing deep learning models. It serves as an open-source textbook, implementation manual, and framework tutorial designed to guide users through the mathematical foundations and practical application of neural networks. The resource provides detailed instructional content on building various model architectures, including convolutional and recurrent neural networks. It includes a dedicated distributed training guide and a learning path that covers the fundamentals of tensors, automatic

This project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip

AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo

Horovod is a distributed deep learning framework designed to scale machine learning training across multiple GPUs and nodes. It functions as an orchestrator for multi-GPU scaling and a tool for distributed gradient averaging, allowing users to increase compute capacity without rewriting core model logic. The project provides a consistent communication interface that supports multi-framework model distribution across TensorFlow, PyTorch, Keras, and MXNet. It leverages an MPI distributed training library to synchronize gradients across processes using collective communication operations. The s

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

AllenNLP is a PyTorch-based research library and deep learning language toolkit designed for developing and training neural network architectures for linguistic tasks. It provides a distributed training system that coordinates data and gradients across multiple GPUs and a framework for integrating pretrained transformer architectures. The system distinguishes itself with a dedicated algorithmic bias mitigation tool used to identify and reduce bias in linguistic model predictions. It also includes model influence analysis to interpret predictions by calculating the influence of specific traini

Sonnet is a modular machine learning framework and TensorFlow neural network library designed for building composable deep learning architectures. It functions as a model orchestrator that manages parameters, state serialization, and graph exports during the training process. The framework provides a distributed training system to synchronize gradients and spread workloads across multiple GPUs or hardware devices. It enables the design of reusable research components through high-level abstractions and subclassing. The library covers neural network architecture design through sequential laye

Sonnet is a modular machine learning framework and TensorFlow library used for building, training, and managing deep learning models. It functions as a system for composing neural networks from reusable modules and layers that encapsulate their own parameters and internal states. The project provides specialized tools for distributed model training, enabling the synchronization of gradients across multiple hardware devices. It also serves as a model state management system, allowing for the persistence of neural network weights and the export of portable models that separate the computation g

Monolith is a distributed recommendation model framework and asynchronous training engine designed to build and train large-scale deep learning architectures. It functions as a distributed model trainer that processes massive datasets across multiple compute nodes using asynchronous update mechanisms. The system features a dedicated embedding table manager that creates unique, feature-isolated tables to prevent representation collisions. It also includes a real-time weight updater to capture immediate changes in user interest and data hotspots through continuous parameter synchronization. Th

BasicSR is a PyTorch-based image restoration toolbox and framework designed for training and deploying deep learning models to upscale, denoise, and deblur images and videos. It serves as a comprehensive system for image super-resolution and video quality restoration, providing the necessary infrastructure to recover fine visual details and increase pixel density. The project distinguishes itself through specialized toolkits for facial image enhancement and high-fidelity face synthesis, as well as a dedicated video quality restoration suite that utilizes deformable convolutions and generative

BELLE is a specialized implementation of Chinese conversational large language models, encompassing a full instruction tuning framework. It provides a pipeline for training, evaluating, and deploying models optimized for natural language understanding and dialogue tasks in the Chinese language. The project is distinguished by its integrated approach to model refinement, combining the curation of multi-million entry instruction datasets with a distributed training pipeline. This pipeline supports both full fine-tuning and low-rank adaptation to optimize conversational performance. The system

This project is a comprehensive educational curriculum and structured learning path covering the full lifecycle of large language models. It provides a guided progression through the theory, architecture, training, and deployment of these models. The curriculum includes specialized guides on transformer architecture, model training tutorials, and frameworks for designing autonomous agents. It also provides dedicated resources for studying model safety and ethics. The material covers a wide range of technical capabilities, including distributed training strategies, parameter-efficient fine-tu

Chainer is an open-source deep learning framework built around define-by-run automatic differentiation, where computation graphs are constructed dynamically during forward execution. This imperative approach allows networks to be built using standard Python control flow, with gradients computed automatically through reverse-mode differentiation on the dynamically recorded graph. The framework supports GPU acceleration through a NumPy-compatible array backend with CUDA and cuDNN support, and provides a pluggable device abstraction that lets users switch between CPU and GPU computation without c

Pythia est un framework de recherche multimodal et un système d'entraînement distribué conçu pour construire, entraîner et évaluer de grands modèles qui combinent des données visuelles et linguistiques. Il fournit un environnement modulaire pour développer des modèles vision-langage, se concentrant sur l'intégration d'entrées d'images et de texte dans des représentations de caractéristiques partagées. Le framework utilise une architecture modulaire qui découple les blocs de construction du modèle en composants interchangeables, permettant une configuration flexible des modules de vision et de langage. Il inclut une suite de benchmarks pour exécuter des modèles de référence contre des jeux de données standardisés afin d'établir des bases de performance cohérentes pour les tâches vision-langage. Le système prend en charge des pipelines d'entraînement distribués pour mettre à l'échelle le développement de modèles sur plusieurs nœuds de calcul et utilise des fichiers de configuration externes pour le mappage des hyperparamètres afin d'assurer la reproductibilité de la recherche.

Flashlight est une bibliothèque de machine learning et de tenseurs autonome en C++ utilisée pour construire et entraîner des réseaux de neurones. Elle fonctionne comme un framework complet de réseaux de neurones et un moteur de différenciation automatique, fournissant les outils pour construire des graphes de calcul et calculer les gradients via la rétropropagation. Le projet sert de framework d'entraînement distribué, utilisant des opérations all-reduce pour synchroniser les gradients et les paramètres sur plusieurs nœuds de calcul et appareils. Il se distingue par une intégration profonde de la manipulation de tenseurs haute performance, l'interopérabilité native de la mémoire des appareils et un système pour synchroniser les poids entre les workers distribués afin d'accélérer l'entraînement de modèles à grande échelle. Le framework couvre un large éventail de capacités de deep learning, incluant la composition modulaire de couches pour concevoir des architectures complexes comme des blocs résiduels et des cellules récurrentes. Il fournit des utilitaires étendus de gestion de données pour l'ingestion et le préchargement, ainsi que des systèmes de sérialisation pour persister les états de modèle. De plus, il inclut une suite d'outils de surveillance et d'observabilité pour suivre les métriques d'entraînement et mesurer les erreurs de séquence. La bibliothèque est implémentée en C++.

Flashlight est une bibliothèque de machine learning en C++ et un framework de deep learning conçu pour construire et entraîner des réseaux de neurones. Il fonctionne comme une bibliothèque de manipulation de tenseurs et un moteur de différenciation automatique qui suit les opérations pour calculer les gradients via la rétropropagation pour l'optimisation des modèles. Le projet se distingue par son rôle de framework d'entraînement distribué, utilisant la synchronisation de gradient all-reduce et des environnements distribués pour mettre à l'échelle les charges de travail de machine learning sur plusieurs nœuds et appareils. Il dispose d'une interface mémoire agnostique au backend et d'une gestion basée sur RAII pour découpler les opérations sur tenseurs du matériel physique. Le framework couvre une large surface de capacités, incluant la construction d'architectures de réseaux de neurones avec des couches convolutionnelles, linéaires et récurrentes. Il fournit des utilitaires étendus pour l'algèbre tensorielle, la gestion et le batching de jeux de données, la sérialisation binaire versionnée pour les états de modèle, et des outils de surveillance pour suivre les métriques d'entraînement et l'utilisation de la mémoire.

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr