Awesome GitHub RepositoriesParallel Data Transformation

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

Genesis World is an embodied AI simulation platform designed for training robotic agents through physics-based interactions. It centers on a multi-physics simulation engine that integrates rigid body, particle, and finite element method dynamics, supported by a parallel simulation kernel compiler that translates Python functions into optimized GPU and CPU kernels. The platform features a photorealistic robot renderer that utilizes path-tracing and Gaussian Splatting to generate synthetic training data. It includes a domain randomization framework to vary lighting and physical parameters acros

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Dask est un framework de calcul parallèle et un planificateur de tâches distribué conçu pour mettre à l'échelle les flux de travail de science des données Python, des machines uniques aux grands clusters. Il fonctionne comme un gestionnaire de ressources de cluster qui orchestre la logique computationnelle en représentant les tâches et leurs dépendances sous forme de graphes acycliques dirigés. Cette architecture permet au système d'automatiser la distribution des charges de travail sur le matériel disponible tout en gérant des exigences d'exécution complexes. Le projet se distingue par un moteur d'évaluation paresseuse qui diffère les opérations sur les données jusqu'à ce qu'elles soient explicitement demandées, permettant une optimisation globale du graphe et une allocation efficace des ressources. Il intègre le déversement de données conscient de la mémoire pour éviter les plantages du système lors du traitement de jeux de données dépassant la mémoire disponible, et il utilise la fusion de graphes de tâches pour combiner des séquences d'opérations en étapes d'exécution uniques, minimisant la surcharge de planification et la communication entre nœuds. La plateforme fournit une surface de capacités complète pour l'analyse de données à grande échelle, incluant le support pour l'apprentissage automatique distribué, l'intégration du calcul haute performance et le traitement de données parallèle. Elle offre des outils étendus pour la gestion du cycle de vie des clusters, le profilage des performances et la surveillance en temps réel de l'exécution des tâches. Les utilisateurs peuvent déployer ces environnements sur diverses infrastructures, incluant le matériel local, les fournisseurs cloud, les systèmes conteneurisés et les clusters de calcul haute performance.

Rayon is a data parallelism library for Rust that provides a framework for converting sequential computations into parallel operations. It enables the transformation of standard data structures and loops into parallel iterators, allowing workloads to be distributed across multiple processor cores. By utilizing a work-stealing scheduler, the library dynamically balances tasks to maximize throughput and minimize execution time. The library distinguishes itself through its focus on safe, scoped task synchronization, which ensures that all spawned operations complete before a scope exits to preve

dbt-core is a command-line framework for transforming data within a warehouse using modular SQL and version control. It functions as a data transformation engine that enables users to define data structures and business logic through declarative configuration files, which the system then compiles into executable code. By managing complex data dependencies through a directed acyclic graph, it ensures that transformation tasks execute in the correct order while maintaining a manifest-driven state to track lineage and execution history. The project distinguishes itself through an adapter-based d

Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

conc is a Go concurrency library and structured concurrency framework providing primitives for managing parallel tasks, mapping slices, and collecting results. It implements a system for spawning scoped tasks to ensure all child processes complete before their parent exits. The library includes a goroutine pool manager to limit active concurrent processes and a panic-safe task runner that catches panics in goroutines and propagates stack traces to the parent. It also provides a concurrent map-reduce tool for transforming data slices and processing streams in parallel while maintaining the ori

PocketFlow is a graph-based framework for designing and executing large language model operations and reasoning patterns. It serves as an orchestrator for building goal-oriented autonomous agents, multi-agent systems, and retrieval-augmented generation pipelines. The system is distinguished by its ability to coordinate autonomous AI agents that use shared memory and tools to solve complex goals, supported by a structured output engine that enforces schema-consistent responses. It utilizes graph-based workflow orchestration to manage sequences of model operations and supports supervisor-based

h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

oneAPI Threading Building Blocks (oneTBB)

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

ArrayFire est un framework de calcul agnostique au matériel et un moteur de tenseurs compilé JIT conçu pour le calcul numérique haute performance. Il sert de bibliothèque de calcul numérique GPU et de toolkit de traitement du signal parallèle qui abstrait les backends matériels, permettant à la même base de code de s'exécuter sur diverses architectures GPU et CPU. Le projet se distingue par un moteur JIT qui utilise la compilation d'expressions pour fusionner les opérations et minimiser la surcharge mémoire. Il emploie un graphe d'exécution différée pour optimiser les chaînes de calcul et fournit des primitives d'interopérabilité pour partager des données et des contextes d'exécution avec des plateformes de calcul externes comme CUDA et OpenCL. La bibliothèque couvre un large éventail de capacités, incluant l'algèbre linéaire parallèle, le traitement du signal numérique et la vision par ordinateur accélérée. Elle fournit des outils pour l'implémentation de l'apprentissage automatique, la simulation de modélisation financière et la résolution d'équations aux dérivées partielles pour les simulations de systèmes physiques. Son système de gestion de tenseurs gère l'allocation de tableaux multidimensionnels, le découpage et les transferts de données hôte-périphérique.

Async is a Swift library that provides a wrapper for Grand Central Dispatch to simplify the management of asynchronous tasks and queues. It functions as a task scheduler for the execution, timing, and orchestration of code blocks. The project includes a task chainer to sequence operations across different threads without nested callback structures and a concurrent loop wrapper that distributes iterations across multiple threads. It also features a token-based cancellation system to track and remove pending operations from the execution queue before they run. The library covers concurrency ma

ndarray est une bibliothèque de tableaux multidimensionnels pour Rust qui sert de framework d'algèbre linéaire et d'outil de calcul scientifique. Elle fournit l'infrastructure de base pour créer et manipuler des tableaux n-dimensionnels, fonctionnant à la fois comme un processeur de tableaux parallèle et une boîte à outils pour l'analyse de données numériques. La bibliothèque se distingue en fournissant un découpage (slicing) et des vues mémoire efficaces, permettant le partage de données sans copie. Elle tire parti de bibliothèques mathématiques backend optimisées pour la multiplication de matrices à haute vitesse et distribue les itérations mathématiques lourdes sur plusieurs threads CPU pour accélérer le traitement. Le projet couvre un large éventail d'opérations mathématiques, notamment l'arithmétique élément par élément, l'agrégation de données basée sur les axes et les calculs de produit scalaire. Elle inclut également des utilitaires complets pour la manipulation de tableaux tels que le remodelage, l'aplatissement, l'empilement et la génération de grilles de coordonnées, ainsi qu'une prise en charge de la génération de tableaux aléatoires et de la sérialisation.