Awesome GitHub RepositoriesParallel Data Transformation

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

Genesis World is an embodied AI simulation platform designed for training robotic agents through physics-based interactions. It centers on a multi-physics simulation engine that integrates rigid body, particle, and finite element method dynamics, supported by a parallel simulation kernel compiler that translates Python functions into optimized GPU and CPU kernels. The platform features a photorealistic robot renderer that utilizes path-tracing and Gaussian Splatting to generate synthetic training data. It includes a domain randomization framework to vary lighting and physical parameters acros

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Codon is an LLVM-based Python compiler and statically typed implementation that translates source code into optimized machine instructions. It functions as a high-performance numerical backend and a GPU computing framework designed to remove runtime overhead. The project implements a compiled alternative to NumPy, translating array logic directly into machine code. It differentiates itself by generating specialized hardware kernels for graphics processors and utilizing static type inference to enable aggressive machine-code optimization. The system provides capabilities for parallel workload

Dask es un framework de computación paralela y un programador de tareas distribuido diseñado para escalar flujos de trabajo de ciencia de datos en Python desde máquinas individuales hasta grandes clústeres. Funciona como un gestor de recursos de clúster que orquesta la lógica computacional representando las tareas y sus dependencias como grafos acíclicos dirigidos. Esta arquitectura permite al sistema automatizar la distribución de cargas de trabajo a través del hardware disponible mientras gestiona requisitos de ejecución complejos. El proyecto se distingue por un motor de evaluación perezosa que difiere las operaciones de datos hasta que se solicitan explícitamente, permitiendo la optimización global del grafo y una asignación eficiente de recursos. Incorpora el volcado de datos consciente de la memoria para evitar fallos del sistema al procesar conjuntos de datos que exceden la memoria disponible, y utiliza la fusión de grafos de tareas para combinar secuencias de operaciones en pasos de ejecución únicos, minimizando la sobrecarga de programación y la comunicación entre nodos. La plataforma proporciona una superficie de capacidades integral para el análisis de datos a gran escala, incluyendo soporte para aprendizaje automático distribuido, integración de computación de alto rendimiento y procesamiento de datos en paralelo. Ofrece herramientas extensas para la gestión del ciclo de vida del clúster, perfilado de rendimiento y monitoreo en tiempo real de la ejecución de tareas. Los usuarios pueden desplegar estos entornos en diversas infraestructuras, incluyendo hardware local, proveedores de nube, sistemas en contenedores y clústeres de computación de alto rendimiento.

Rayon is a data parallelism library for Rust that provides a framework for converting sequential computations into parallel operations. It enables the transformation of standard data structures and loops into parallel iterators, allowing workloads to be distributed across multiple processor cores. By utilizing a work-stealing scheduler, the library dynamically balances tasks to maximize throughput and minimize execution time. The library distinguishes itself through its focus on safe, scoped task synchronization, which ensures that all spawned operations complete before a scope exits to preve

dbt-core is a command-line framework for transforming data within a warehouse using modular SQL and version control. It functions as a data transformation engine that enables users to define data structures and business logic through declarative configuration files, which the system then compiles into executable code. By managing complex data dependencies through a directed acyclic graph, it ensures that transformation tasks execute in the correct order while maintaining a manifest-driven state to track lineage and execution history. The project distinguishes itself through an adapter-based d

Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

conc is a Go concurrency library and structured concurrency framework providing primitives for managing parallel tasks, mapping slices, and collecting results. It implements a system for spawning scoped tasks to ensure all child processes complete before their parent exits. The library includes a goroutine pool manager to limit active concurrent processes and a panic-safe task runner that catches panics in goroutines and propagates stack traces to the parent. It also provides a concurrent map-reduce tool for transforming data slices and processing streams in parallel while maintaining the ori

PocketFlow is a graph-based framework for designing and executing large language model operations and reasoning patterns. It serves as an orchestrator for building goal-oriented autonomous agents, multi-agent systems, and retrieval-augmented generation pipelines. The system is distinguished by its ability to coordinate autonomous AI agents that use shared memory and tools to solve complex goals, supported by a structured output engine that enforces schema-consistent responses. It utilizes graph-based workflow orchestration to manage sequences of model operations and supports supervisor-based

h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

oneAPI Threading Building Blocks (oneTBB)

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

ArrayFire es un framework de computación agnóstico al hardware y un motor de tensores compilado JIT diseñado para la computación numérica de alto rendimiento. Sirve como una biblioteca de computación numérica en GPU y un kit de herramientas de procesamiento de señales paralelo que abstrae los backends de hardware, permitiendo que el mismo código base se ejecute en diversas arquitecturas de GPU y CPUs. El proyecto se distingue por un motor JIT que utiliza la compilación de expresiones para fusionar operaciones y minimizar la sobrecarga de memoria. Emplea un grafo de ejecución diferida para optimizar las cadenas de cálculo y proporciona primitivas de interoperabilidad para compartir datos y contextos de ejecución con plataformas de computación externas como CUDA y OpenCL. La biblioteca cubre una amplia gama de capacidades, incluyendo álgebra lineal paralela, procesamiento digital de señales y visión artificial acelerada. Proporciona herramientas para la implementación de aprendizaje automático, simulación de modelos financieros y la resolución de ecuaciones diferenciales parciales para simulaciones de sistemas físicos. Su sistema de gestión de tensores maneja la asignación de matrices multidimensionales, el corte (slicing) y las transferencias de datos entre host y dispositivo.

Async es una librería Swift que proporciona un wrapper para Grand Central Dispatch para simplificar la gestión de tareas y colas asíncronas. Funciona como un planificador de tareas para la ejecución, temporización y orquestación de bloques de código. El proyecto incluye un encadenador de tareas para secuenciar operaciones a través de diferentes hilos sin estructuras de callback anidadas y un wrapper de bucle concurrente que distribuye iteraciones a través de múltiples hilos. También cuenta con un sistema de cancelación basado en tokens para rastrear y eliminar operaciones pendientes de la cola de ejecución antes de que se ejecuten. La librería cubre la gestión de concurrencia mediante la coordinación de grupos asíncronos y la sincronización de múltiples tareas para activar acciones tras la finalización colectiva.

ndarray es una biblioteca de arreglos multidimensionales para Rust que sirve como framework de álgebra lineal y herramienta de computación científica. Proporciona la infraestructura central para crear y manipular arreglos de n-dimensiones, funcionando tanto como un procesador de arreglos paralelo como un kit de herramientas para el análisis de datos numéricos. La biblioteca se distingue por proporcionar cortes (slicing) y vistas de memoria eficientes, lo que permite compartir datos sin copiarlos. Aprovecha bibliotecas matemáticas de backend optimizadas para la multiplicación de matrices de alta velocidad y distribuye iteraciones matemáticas pesadas a través de múltiples hilos de CPU para acelerar el procesamiento. El proyecto cubre una amplia gama de operaciones matemáticas, incluyendo aritmética elemento a elemento, agregación de datos basada en ejes y cálculos de producto punto. También incluye utilidades integrales para la manipulación de arreglos como el cambio de forma, aplanamiento, apilamiento y generación de cuadrículas de coordenadas, junto con soporte para la generación de arreglos aleatorios y serialización.