Awesome GitHub RepositoriesDistributed Parameter Sharding

TensorFlow is a comprehensive machine learning framework designed for the construction, training, and deployment of complex mathematical models. It utilizes a graph-based execution model that represents operations as directed acyclic graphs, enabling automatic differentiation and efficient parallel processing. The system provides high-level interfaces for defining neural network architectures, alongside a robust engine for managing multidimensional array structures and tensor mathematics. The framework distinguishes itself through a scalable distributed runtime that orchestrates workloads acr

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

Llama 3 is a collection of pretrained, autoregressive transformer-based models designed for natural language generation, reasoning, and complex instruction following. It functions as a generative AI framework that provides the infrastructure for managing model weights, executing neural network inference, and handling computational workloads across diverse knowledge domains. The project distinguishes itself through an integrated AI safety toolkit that employs secondary classification filtering to inspect inputs and outputs, ensuring adherence to usage compliance and safety standards. It suppor

This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en

Mamba is a deep learning framework designed for building and training sequence models that process long-range data dependencies with linear-time computational efficiency. By utilizing selective state space modeling, the library enables the construction of neural network architectures that replace traditional attention mechanisms with high-performance state space operations. The framework distinguishes itself through the use of data-dependent state gating, which allows the model to dynamically filter information flow based on the input sequence. To ensure high throughput, it incorporates hardw

Flax is a deep learning framework and JAX neural network library designed for building complex machine learning models. It functions as a distributed training library and model state manager, providing a toolkit for defining flexible neural network architectures and scaling their training across multiple hardware devices. The project is characterized by a design that separates network logic from parameter values to remain compatible with pure functions. It uses hierarchical module composition to organize networks as trees of nested modules and employs a reference-based state management system

Gemma es una familia de modelos de lenguaje de gran tamaño de pesos abiertos basados en una arquitectura transformer solo de decodificador. Estos modelos están diseñados para la generación de texto y conversaciones multimodales, capaces de procesar y generar respuestas basadas tanto en secuencias de entrada textuales como visuales. El proyecto proporciona un modelo de IA ajustable (fine-tunable) que admite el ajuste de pesos y la adaptación de bajo rango (LoRA) para especializar el rendimiento en tareas particulares. Incluye soporte para pesos cuantizados para reducir el uso de memoria y aumentar la velocidad de inferencia en hardware limitado. La superficie de capacidades cubre la integración de IA multimodal, optimización de memoria mediante fragmentación de parámetros (sharding) y la integración de herramientas y APIs externas para recuperar datos en tiempo real. Permite además la generación de imágenes a partir de texto y el muestreo de salidas de texto estructurado.

trlx es una biblioteca de aprendizaje por refuerzo y framework de entrenamiento diseñado para alinear modelos de lenguaje grandes utilizando retroalimentación humana. Sirve como un entrenador distribuido y orquestador de cómputo para escalar modelos de alto número de parámetros a través de múltiples GPUs y nodos. El proyecto proporciona herramientas para el aprendizaje por refuerzo a partir de retroalimentación humana y alineación de modelos. Implementa optimización basada en modelos de recompensa y optimización de política proximal para refinar el comportamiento del modelo basado en recompensas orientadas a objetivos o conjuntos de datos etiquetados por humanos. El framework cubre estrategias de entrenamiento distribuido, incluyendo paralelismo de modelos, fragmentación de parámetros y sincronización de gradientes entre múltiples nodos. También incorpora restricciones como la divergencia KL para gestionar la deriva del modelo durante el proceso de aprendizaje por refuerzo.

This project is a distributed machine learning platform and sparse deep learning framework designed for training and serving models with high-dimensional sparse data. It functions as an online model serving infrastructure and recommendation system engine, enabling real-time item retrieval and scoring using deep tree matching and neural networks. The system distinguishes itself through a multi-task learning framework that optimizes multiple objective functions within a shared representation space. It features a specialized online serving infrastructure that supports dynamic model hot-loading a

Higgsfield is a distributed machine learning training framework and GPU cluster orchestrator designed for scaling neural networks with billions of parameters. It functions as a large model sharding system and a containerized deployment tool to manage computational workflows across heterogeneous compute resources. The platform provides a centralized interface for experiment management, enabling the monitoring of real-time telemetry, performance metrics, and logs. It ensures reproducible results by using container isolation to standardize dependencies across different computing environments. T