5 repository-uri
Libraries and frameworks for building distributed, fault-tolerant, and event-driven system services.
Explore 5 awesome GitHub repositories matching part of an awesome list · Service Programming. Refine with filters or upvote what's useful.
Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t
Manages complex batch job pipelines with dependency resolution and workflow management.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Lightweight ETL framework bridging the gap between scripts and complex orchestrators.
Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.
Provides libraries for processing LZOP-compressed data.
Spring XD makes it easy to solve common big data problems such as data ingestion and export, real-time analytics, and batch workflow orchestration
Distributed system for data ingestion, real-time analytics, and batch processing.
Serverless proxy for Spark cluster
Exposes analytics jobs and models as real-time or batch services.