apache/shardingsphere-elasticjob
Shardingsphere Elasticjob
ShardSphere-ElasticJob is a Java-based distributed scheduling framework designed to manage workloads across multiple nodes. It provides a system for splitting scheduled tasks into shards and distributing them across a cluster to achieve high-throughput execution.
The framework includes a distributed task failover system that detects node failures and automatically reassigns missed job executions to healthy cluster instances. It also features a cluster resource manager to dynamically allocate execution resources based on system load and capacity.
The system covers high-availability task execution, distributed job sharding, and unified task orchestration for diverse job formats such as scripts and network requests. Observability is provided through job execution monitoring and a central console for tracking events and managing registry centers.
Features
- Job Sharding - Splits single logical jobs into multiple parallel shards distributed across available worker instances for high-throughput execution.
- Distributed Task Schedulers - Orchestrates and distributes complex scheduled tasks across computing clusters via sharding.
- Distributed Task Schedulers - Provides a framework for splitting scheduled tasks into shards and dispatching them across distributed environments.
- High Availability Job Execution - Ensures scheduled jobs continue to run by automatically failing over tasks from crashed nodes.
- Job Scheduling - Manages, monitors, and executes distributed background processing jobs across a cluster.
- Task Failover - Detects node failures and automatically reassigns missed job executions to healthy cluster instances.
- Cluster State Coordinators - Uses Zookeeper to maintain a consistent view of global cluster topology and worker registration for task assignment.
- Node Failure Detection - Implements node health monitoring via heartbeats to trigger automatic task reassignment when workers fail.
- Task Execution Failover - Detects node failures and automatically reassigns missed job executions to healthy cluster instances.
- Task Orchestrators - Provides a centralized interface for managing and executing sequences of diverse background operations.
- Dynamic Shard Redistribution - Adjusts the distribution of job shards across the cluster in real time based on active node availability.
- Resource Allocation - Manages the allocation of execution resources to jobs and dynamically adjusts them based on system load.
- Cluster Resource Managers - Dynamically allocates and scales compute resources for distributed jobs based on current capacity.
- Unified Task Interfaces - Wraps diverse execution formats, such as scripts and network requests, into a consistent model for the scheduler.
- Background Job Monitoring - Provides a central console for tracking the health, status, and failure metrics of distributed asynchronous tasks.
- Cluster Monitoring - Offers a central console for tracking the status of distributed tasks and monitoring cluster health.
