Open-source alternatives to Taskflow

Bend is a high-level parallel programming language and compiler designed to execute code across multi-core CPUs and GPUs automatically. By translating functional source code into a graph-based intermediate representation, it enables massive parallel execution without requiring manual management of threads, locks, or atomic operations. The runtime operates as an interaction net engine, where computations are represented as networks of nodes that reduce through local rewriting rules. This model utilizes a work-stealing scheduler to distribute tasks across thousands of hardware threads, ensuring

AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo

Crossbeam is a concurrency toolkit for Rust providing low-level primitives for writing multi-threaded programs. It focuses on lock-free data structures and memory management primitives designed for shared-memory concurrent environments. The project includes a work-stealing scheduler that uses double-ended queues to balance workloads across multiple processor cores. This system enables the implementation of work-stealing deques to distribute tasks and prevent bottlenecks. The toolkit covers broader capabilities for parallel algorithm development, multi-threaded task scheduling, and general co

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

A header-only C++ library for task concurrency

Workflow is an asynchronous C++ task engine designed for building distributed systems and high-performance network services. It provides a framework for orchestrating complex sequences of network, file, and computational operations, allowing developers to compose these tasks into parallel workflows. The library functions as a toolkit for implementing scalable servers and clients for protocols such as HTTP, Redis, MySQL, and Kafka. It distinguishes itself through a task-based concurrency model that manages non-blocking operations and coordinates service discovery, load balancing, and traffic r

This project is a comprehensive collection of reference materials, including a language cheatsheet, a standard library reference, and a concurrency reference. It serves as a guide to modern C++ development, focusing on language syntax, standard library utilities, and template metaprogramming patterns. The repository provides specific guidance on template metaprogramming through a dedicated guide covering compile-time evaluation, type deduction, and variadic template execution. The materials cover a broad range of capabilities, including asynchronous programming, memory management, and system

Numba is a just-in-time compiler that translates high-level Python functions into optimized machine code at runtime. By leveraging the LLVM compiler infrastructure, it provides a framework for accelerating numerical data processing and mathematical computations, enabling performance levels comparable to statically compiled languages. The project distinguishes itself through its ability to perform type-inference-based specialization, which generates machine instructions tailored to the specific data types used during execution. It employs a lazy compilation pipeline that defers translation unt

oneTBB is a C++ parallelism library and framework designed to add multi-core parallelism to applications. It provides a task-based parallelism model that maps logical computational tasks to available hardware cores to eliminate the need for manual thread management. The library functions as a multi-core scaling tool, utilizing generic templates to scale data-parallel operations across processors for portable performance. It employs a task-based framework to ensure computational workloads are distributed across hardware resources. The project covers shared memory parallelism, multi-core task

ThreadPool is a C++ thread management library designed to execute asynchronous tasks using a fixed number of background worker threads. It functions as a concurrent task executor that reduces the overhead associated with the repeated creation and destruction of threads. The library utilizes a synchronized queue to distribute workloads across multiple CPU cores and employs variadic templates to accept any callable function with arbitrary arguments. The system manages concurrency through mutex-protected shared state and condition-variable signaling to wake idle threads. It also provides a mech

Dag-factory is a framework for constructing and managing Apache Airflow data pipelines through declarative configuration files. By replacing manual procedural code with structured YAML definitions, it enables the programmatic generation of complex workflow structures, task dependencies, and execution schedules. The project distinguishes itself by mapping configuration keys directly to Python class constructors and operators, allowing for the dynamic instantiation of objects and custom logic. It supports hierarchical configuration inheritance to standardize settings across environments and pro

Grid.js is a framework-agnostic JavaScript library for rendering interactive data grids. It allows for the display of structured information in tabular formats across different frontend environments, supporting data population from static arrays or JSON imports. The library features a plugin system for extending user interface components and logic, as well as a custom data pipeline for transforming information before it is displayed. It includes built-in support for multilingual localization to translate interface elements and messages. The project covers core data visualization capabilities

Hatchet is an open-source durable workflow engine and task orchestration platform. It provides a framework for building and executing fault-tolerant, multi-step pipelines as directed acyclic graphs (DAGs), with automatic retries, scheduling, and real-time observability. The system is built around durable task checkpointing, which persists execution state after each step so work can resume from the last checkpoint after a worker crash or restart, and it supports event-driven task resumption that pauses a task until a matching external event arrives. The platform distinguishes itself through it

CuPy is a CUDA array computing library that implements a NumPy-compatible interface for executing array operations and numerical computing on NVIDIA GPUs. It serves as a GPU-accelerated numerical library and a CUDA-based SciPy implementation, offloading heavy calculations to graphics hardware to increase processing speed for scientific and engineering workloads. The library enables multi-framework tensor exchange, allowing data buffers to be shared between different deep learning frameworks using standardized memory layouts to avoid memory copies. It also supports custom GPU kernel integratio

Task is a YAML-based task runner and build tool used to define and automate development workflows. It functions as a dependency-based build system and cross-platform task automator, allowing users to execute shell commands across different operating systems using a declarative configuration file. The project operates as an incremental build tool, utilizing file fingerprints and checksums to track state and avoid redundant work by determining if tasks are up to date. It manages execution via a dependency graph to ensure prerequisites are completed before target commands run. The system includ

Thrust is a C++ parallel algorithms library that provides a suite of standard-library-inspired interfaces for execution on multi-core and accelerator hardware. It serves as a CUDA-accelerated data library and a generic parallel programming interface designed to enable high-performance data processing across GPUs and CPUs. The project implements a portable abstraction layer that allows for heterogeneous computing workflows, enabling the same core algorithm logic to run on different hardware accelerators. This is achieved through a generic programming policy design and a backend-agnostic execut

HVM2 is a high-performance execution environment for pure functional programs, implemented as a systems-level runtime in Rust. It functions as a massively parallel functional runtime that uses interaction combinators to achieve automatic parallelism across multi-core CPUs and GPUs. The project distinguishes itself by using a graph-rewriting computational model to execute programs via local reduction rules, which eliminates the need for manual locks or atomic operations. It employs beta-optimal reduction and lazy evaluation to optimize higher-order functions and eliminate redundant computation

Archon is an artificial intelligence agent automation engine designed to orchestrate complex development workflows. It functions as a platform for chaining multi-step tasks into directed graphs, allowing developers to standardize and execute repeatable coding patterns through declarative configuration files. The system distinguishes itself by maintaining stateful context across long-running sessions and executing operations within isolated, containerized worktrees to prevent file interference. It integrates with external language models and provides a centralized registry for sharing and inst

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

This project is a comprehensive suite of AI tools and frameworks, featuring an LLM multi-agent orchestrator, an autonomous agent runtime, and a stateful application framework. It provides the infrastructure to build and manage specialized AI agents capable of coordinating complex tasks through graph-based workflows and shared state. The system is distinguished by its implementation of the Model Context Protocol, allowing for standardized resource discovery and communication between AI clients and servers. It further includes an AI-powered documentation generator designed to analyze source cod

MediaPipe is a cross-platform machine learning framework designed for deploying vision, audio, and text processing models across mobile, desktop, and web environments. It functions as an on-device inference engine that executes complex models locally on edge hardware, ensuring low latency and privacy without requiring a constant internet connection. The framework utilizes a graph-based pipeline orchestration system where data flows through a directed network of modular calculators to ensure synchronized and deterministic processing. It distinguishes itself through a unified runtime that provi

This project is a comprehensive performance programming guide and reference for the Go language, focusing on runtime efficiency and memory optimization. It provides a collection of patterns and techniques designed to increase execution speed by reducing garbage collection overhead and optimizing memory usage. The resource distinguishes itself through detailed reference implementations for memory optimization, such as escape analysis, object pooling, and structure memory alignment. It offers specific strategies for reducing binary size and improving CPU cache efficiency through structure memor

Concurrent Ruby is a comprehensive concurrency toolkit for the Ruby language that provides thread-safe data structures, synchronization primitives, and asynchronous execution patterns. It implements core concurrency abstractions including an actor model framework where isolated actors communicate through asynchronous message passing, a future and promise system for composing non-blocking operations, and thread pool executors that manage reusable worker threads for concurrent task execution. The library distinguishes itself through a broad set of coordination mechanisms that go beyond basic th

Parallel is a Ruby library and multi-process execution framework designed to accelerate CPU-intensive operations. It functions as a parallel job orchestrator and concurrent task runner that enables the execution of code across multiple processes or threads. The project distinguishes itself through secure inter-process communication, utilizing signed data serialization to prevent the injection of forged payloads between parent and child processes. It further differentiates its worker management by assigning unique identifiers to individual processes to prevent collisions when accessing shared

HIP is a C++ GPU kernel language and cross-platform runtime designed for writing portable high-performance compute applications. It provides a programming interface that allows a single source codebase to execute on both AMD and NVIDIA GPU architectures. The project functions as a compatibility layer that enables the conversion and migration of existing CUDA source code to run on AMD hardware. This is achieved through a syntax mapping that mirrors CUDA and a source-to-source translation process during compilation. The toolkit covers the broader surface of cross-platform GPGPU development, in

The project provides high-performance concurrency, enabling highly parallel computation.