1 repo
Awesome GitHub RepositoriesDistributed Task Queues
Infrastructure for distributing and executing background tasks across multiple worker nodes.
Distinguishing note: Focuses on horizontal scaling of task execution rather than message queuing protocols.
Explore 1 awesome GitHub repository matching devops & infrastructure · Distributed Task Queues. Refine with filters or upvote what's useful.
Awesome Distributed Task Queues GitHub Repositories
Find the best repos with AI.We'll search the best matching repositories with AI.
- apache/airflow
apache/airflow
44,326View on GitHubApache Airflow is a platform designed to programmatically author, schedule, and monitor complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of data processing tasks by defining them as code-based directed acyclic graphs. This approach allows for dynamic task generation and precise control over execution dependencies across distributed computing environments. The system is distinguished by its infrastructure-agnostic scheduler and a modular provider-based framework that decouples core orchestration logic from external service integrations. By utilizi
A central scheduler dispatches task instances to a pool of distributed workers to enable horizontal scaling across multiple nodes.
Pythonairflowapacheapache-airflow
