This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It provides a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams. The library distinguishes itself through a comprehensive toolkit for shell-like operations, including a text processing engine for regular expression filtering and frequency analysis, a filesystem utility toolkit for recursive search and path manipulation, and an integrated HTTP client wrapper for building data pipelines that fetch web content. The capability s
Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data pipelines. At its core, it provides a web-based visual dataflow designer where users build directed graphs of processors to route, transform, and mediate data movement between any source and destination without writing custom code. The system records fine-grained data provenance for every data item from ingestion to delivery, supporting audit, debugging, and replay of data lineage. The platform distinguishes itself through a zero-master cluster architecture that distributes proc
Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep
Orchest is a data pipeline orchestrator and containerized workflow manager. It provides a platform for designing, scheduling, and executing complex data processing sequences through a combination of a graphical interface and scripting. The platform distinguishes itself by using containers to manage software dependencies, ensuring consistent execution across different environments. It features a polyglot task scheduler capable of triggering jobs written in multiple programming languages and includes a version control system that tracks historical snapshots of project configurations and code.
Streem ist eine streambasierte Programmiersprache und ein Daten-Pipeline-Orchestrator. Sie bietet eine domänenspezifische Sprache zur Definition paralleler Datenflüsse, die es Benutzern ermöglicht, Datenquellen über eine Sequenz von Operationen, die einzelne Streamelemente transformieren und filtern, mit Zielen zu verknüpfen.
Die Hauptfunktionen von matz/streem sind: Data Pipeline Orchestration, Stream-Based Languages, Collection Element Transformations, Concurrent Data Processors, Concurrent Stream Processing, Data Flow Definitions, Data Pipeline Orchestrators, Stream and Pipeline Orchestration.
Open-Source-Alternativen zu matz/streem sind unter anderem: bitfield/script — This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It… apache/nifi — Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data… prefecthq/prefect — Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as… orchest/orchest — Orchest is a data pipeline orchestrator and containerized workflow manager. It provides a platform for designing,… iterative/dvc — DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models.… holistics/dbml — DBML is a domain-specific language and schema definition language used for documenting database architecture and…