Awesome GitHub RepositoriesParallel Programming Frameworks

DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special

Taichi is a domain-specific programming language embedded in Python designed for high-performance numerical computing and computer graphics. It functions as a parallel compiler that translates high-level mathematical expressions into optimized machine instructions, enabling developers to write compute-intensive algorithms that execute across diverse hardware architectures, including CPUs, GPUs, and specialized accelerators. The project distinguishes itself through a hardware-agnostic execution layer that maps parallel operations to multiple backends such as CUDA, Metal, and Vulkan. By utilizi

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

ZLUDA is a middleware and translation engine designed to enable the execution of unmodified proprietary compute binaries on non-native graphics hardware. It functions as a compatibility layer that bridges vendor-specific compute interfaces with open standards, allowing software originally restricted to a single hardware ecosystem to operate on alternative graphics processing units. The project achieves this through a combination of dynamic library interception and runtime instruction translation. By replacing standard system libraries and mapping proprietary compute calls to open standards, t

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Highway एक पोर्टेबल C++ लाइब्रेरी और हार्डवेयर एब्स्ट्रैक्शन लेयर है जिसे सिंगल इंस्ट्रक्शन मल्टीपल डेटा (SIMD) कोड लिखने के लिए डिज़ाइन किया गया है। यह एक एकीकृत इंटरफेस प्रदान करता है जो डेटा-पैरेलल लॉजिक को विभिन्न CPU इंस्ट्रक्शन सेटों पर मैप करता है, जिससे उच्च-प्रदर्शन वाले सॉफ्टवेयर का विकास संभव होता है जो आर्किटेक्चर-विशिष्ट असेंबली की आवश्यकता के बिना विभिन्न प्रोसेसर आर्किटेक्चर पर चलता है। प्रोजेक्ट में एक डायनामिक इंस्ट्रक्शन डिस्पैचर है जो पता लगाए गए हार्डवेयर के आधार पर रनटाइम पर सबसे कुशल CPU इंस्ट्रक्शन सेट का चयन करता है। यह स्टेटिक टारगेट स्पेशलाइजेशन और नए हार्डवेयर टारगेट या कस्टम SIMD ऑपरेशंस को जोड़ने के लिए एक्स्टेंसिबल तंत्र का भी समर्थन करता है।

HIP is a C++ GPU kernel language and cross-platform runtime designed for writing portable high-performance compute applications. It provides a programming interface that allows a single source codebase to execute on both AMD and NVIDIA GPU architectures. The project functions as a compatibility layer that enables the conversion and migration of existing CUDA source code to run on AMD hardware. This is achieved through a syntax mapping that mirrors CUDA and a source-to-source translation process during compilation. The toolkit covers the broader surface of cross-platform GPGPU development, in

An Open Source Implementation of the Actor Model in C++

The C++ Standard Library for Parallelism and Concurrency

cuda-oxide is an experimental Rust-to-CUDA compiler that lets you write (SIMT) GPU kernels in safe(ish), idiomatic Rust. It compiles standard Rust code directly to PTX — no DSLs, no foreign language bindings, just Rust.

ISPC is a vectorizing compiler and SIMD parallel programming language that implements a single program multiple data model. It serves as a toolchain for translating C-based code with parallel extensions into optimized machine code for various CPU and GPU architectures using an LLVM backend. The compiler is designed for cross-platform SIMD toolchain support, generating specialized instruction sets for x86 SSE/AVX, ARM NEON, and Intel GPU from a single source. It features a runtime dispatch mechanism that selects the most efficient hardware-specific implementation for the current system during

Kokkos C++ Performance Portability Programming Ecosystem: The Programming Model - Parallel Execution and Memory Abstraction

General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)

RaftLib is an open-source C++ Library that provides a framework for implementing parallel and concurrent data processing pipelines. It is designed to simplify the development of high-performance data processing applications by abstracting away the complexities of parallelism, concurrency, and…

Copyright 2024 NVIDIA Corporation

Legion is a parallel programming model for distributed, heterogeneous machines.

A header-only C++ library for task concurrency