1 repository
Awesome GitHub RepositoriesData Processing Script Runners
Environments for executing data processing scripts with configurable environment variables and parameters.
Distinct from Dynamic Script Execution Environments: Distinct from dynamic script execution: focuses on data pipeline script execution rather than document rendering.
Explore 1 awesome GitHub repository matching development tools & productivity · Data Processing Script Runners. Refine with filters or upvote what's useful.
Awesome Data Processing Script Runners GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- spotify/luigi
spotify/luigi
18,676Vezi pe GitHubLuigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t
Runs data processing scripts within a pipeline by configuring environment variables and job properties.
Pythonhadoopluigiorchestration-framework
