How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t
Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.
Spring XD makes it easy to solve common big data problems such as data ingestion and export, real-time analytics, and batch workflow orchestration
The main features of spring-projects/spring-xd are: Service Programming.
Projects with overlapping indexed features include: hydrospheredata/mist — Serverless proxy for Spark cluster. mara/data-integration — A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow. spotify/luigi — Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a… twitter/elephant-bird — Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.
