WeiYe-Jing/datax-web
Datax Web
DataX Web is a web-based management platform for scheduling, building, executing, and monitoring distributed data synchronization jobs powered by DataX. It provides a visual console for creating and managing DataX tasks without manual JSON configuration, with a distributed executor cluster that auto-registers worker nodes and supports configurable routing and blocking strategies for task distribution.
The platform offers cron-based task scheduling with dynamic start, stop, and immediate status changes, along with incremental sync capabilities that pass dynamic parameters to extract only new or changed records during scheduled runs. It includes role-based access control with admin and normal user tiers, real-time log streaming to a web console for monitoring running tasks, and email-based failure notifications with a pluggable extension interface for additional alert channels.
Beyond DataX tasks, the system supports executing Shell, Python, and PowerShell scripts through the same scheduling interface. It provides web-based configuration for data source connections with testing, task template creation, and JVM parameter customization, along with execution reports showing success and failure distributions.
Features
- Tasks and Scheduling - Support cron-based scheduling with dynamic start, stop, and status changes, including termination of running tasks that take effect immediately.
- Dynamic Parameter Injections - Passes dynamic field values and intervals to DataX tasks for incremental data extraction.
- Scheduled Incremental Sync Configurations - Defines parent-child task dependencies and configures incremental sync with auto-calculated date ranges.
- Scheduled Sync Engines - Automates periodic extraction of only new or changed records from source databases using dynamic parameter injection.
- Multi-Source Data Aggregation - Connects and manages diverse data sources like RDBMS, Hive, HBase, and MongoDB through a unified web console.
- Multi-Source Data Integration - Generates column information and simplifies configuration for multiple data sources.
- Incremental Data Exporters - Extracts only new or changed records based on a tracked cursor or timestamp for each scheduled run.
- Incremental Data Synchronization - Runs data synchronization tasks that only pull new or changed records since the last execution.
- Data Synchronization Consoles - A web interface for creating, scheduling, and monitoring DataX data synchronization tasks across multiple data sources.
- Data Sync Job Builders - Builds DataX synchronization JSON configurations through a web interface, storing them in a database.
- Cron Scheduling - Triggers data synchronization tasks on a cron schedule with dynamic start and stop.
- Distributed Job Executors - Distributes task execution across a cluster of auto-registering executors with routing strategies.
- DataX Distributed Executors - A cluster of worker nodes that execute DataX tasks with configurable routing, blocking, and retry strategies.
- Task Timeout and Retry Configurations - Set custom timeout and retry counts for tasks, with automatic interruption on timeout and email-based failure alerts that can extend to other channels.
- Task & Job Management - Provides a web interface to create, configure, and monitor DataX data synchronization jobs without writing code.
- Cron-Based - Support cron-based scheduling, dynamic task status changes, start/stop operations, and termination of running tasks with immediate effect.
- Distributed Job Execution - Distributes DataX job execution across multiple worker nodes to handle large-scale data synchronization workloads.
- Visual Job Builders - Builds DataX synchronization jobs through a web interface without writing JSON configuration files manually.
- Recurring Job Scheduling - Run DataX tasks on a recurring schedule, automating data transfers between sources and targets.
- Cron Job Automation - Schedules recurring data synchronization jobs with dependency chains, retry logic, and timeout controls.
- DataX Job Schedulers - A cron-based scheduler that automates DataX data synchronization tasks with distributed execution and real-time monitoring.
- Web-Based Job Management - Generates DataX JSON configuration files through a web interface for job management.
- User Role Management - Administers system users with admin and normal roles for role-based access control.
- Role-Based Access Control - Manages system access with admin and normal user roles controlling feature visibility.
- Web-Based Configuration Generators - Generates DataX JSON configuration files through a web interface for job management.
- Clustered Task Distribution - Assigns tasks to multiple executors that auto-register with a central scheduler for cluster deployment.
- Pluggable Data Source Architectures - Connects to diverse data sources like RDBMS, Hive, HBase, and MongoDB through a pluggable adapter interface.
- Automatic Task Retries - Set custom timeout and retry counts for tasks, automatically interrupting or retrying failed jobs based on configuration.
- Task Execution Monitoring - Monitor the real-time execution logs and status of data synchronization tasks through a visual web interface.
- Live Log Streaming - Streams live execution logs and sends failure notifications via email with extensible alert channels.
- Real-Time Application Log Monitoring - Streams live execution logs to a web console for monitoring and terminating running tasks.
- Incremental Sync Configurations - Configures incremental fields and partition parameters to automatically compute data ranges for each scheduled run.
- Incremental Update Configurations - Configures incremental fields and retry logic to automatically compute data intervals for scheduled tasks.
- Data Source Connectivity Tools - Verifies data source configurations with a manual test button for JDBC sources.
- Plugin-Based Data Source Adapters - A plugin-based adapter that connects DataX to RDBMS, Hive, HBase, ClickHouse, MongoDB, and other data sources.
- Scheduled Execution Reports - Provide real-time operational data and scheduling reports, including date distribution charts for successful and failed executions.
- Routing and Blocking Policies - Provides routing strategies and blocking policies for managing concurrent task execution across a cluster.
- Parent-Child Task Triggers - Configure child tasks that trigger automatically after a parent task completes successfully, supporting comma-separated child task lists.
- Configurable Routing Strategies - Distributes job execution across a cluster of executors using configurable routing strategies like round-robin, random, or failover.
- Task Schedulers - Extend task types beyond DataX to include Shell, Python, and PowerShell scripts, all managed through the same scheduling interface.
- Script Execution Tasks - Extend task types beyond DataX to include Shell, Python, and PowerShell scripts, all managed through the same scheduling system.
- Configurable Blocking Strategies - Applies configurable blocking strategies when the scheduler dispatches tasks faster than the executor can process.
- Partition Parameter Passings - Passes partition field values as runtime parameters so DataX tasks target the correct table partitions during execution.
- Parent-Child Dependencies - A system for defining parent-child task dependencies and triggering child tasks automatically after parent completion.
- Task Templates - Builds reusable templates from common JSON configurations for creating new jobs.
- Email Alert Notifications - Sends failure notifications via email with a pluggable interface for additional alert channels.
- Scheduled Execution Reports - Provide real-time operational data and scheduling reports, including distribution charts for successful and failed executions.
- Reusable Task Templates - Builds reusable templates from common JSON configurations for creating new jobs.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
