Open-source alternatives to Data Integration

High throughput workflows and automation for HPC

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments

An engine for managing the execution of container-based workflows.

Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core focus on codebase indexing and knowledge graph extraction. The engine uses a function-graph execution model where user-defined Python functions are composed into a directed acyclic graph, and it processes data incrementally so only changed source records or code paths are re-computed, avoiding full recomputation at any scale. It supports automatic schema inference from transformation pipeline type annotations and provides full data lineage tracing, tagging every output record wi

The official repository for the AiiDA code

Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Dask is a parallel computing framework and distributed task scheduler designed to scale Python data science workflows from single machines to large clusters. It functions as a cluster resource manager that orchestrates computational logic by representing tasks and their dependencies as directed acyclic graphs. This architecture allows the system to automate the distribution of workloads across available hardware while managing complex execution requirements. The project distinguishes itself through a lazy evaluation engine that defers data operations until they are explicitly requested, enabl

dynamic DAG workflows

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

A framework (comand line tool libraries) for creating flexible compute pipelines

GNU-Make-like utility for managing builds and complex workflows

Data workflow tool, like a "Make for data"

Workflows for Fission: Fast, reliable and lightweight function composition for serverless functions

A scala based DSL and framework for writing and executing bioinformatics pipelines as Directed Acyclic GRaphs