Awesome GitHub RepositoriesData Processing Tasks

Kestra is a declarative workflow orchestrator designed to manage complex task dependencies and automated processes through versioned configuration files. It functions as a distributed platform that decouples task scheduling from execution by offloading computational workloads to a fleet of worker nodes. The system uses a reactive, event-driven engine to initiate workflows automatically in response to external signals, webhooks, schedules, or file system changes. The platform distinguishes itself through a modular plugin architecture that allows for the integration of custom tasks and external

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

CVAT is an open-source, web-based platform designed for annotating images, videos, and 3D point clouds to create high-quality training datasets for machine learning. It functions as a containerized server that orchestrates the entire lifecycle of computer vision data, from initial task creation and manual labeling to quality assurance and final dataset export. The platform distinguishes itself through deep integration with machine learning models, allowing users to deploy custom AI models as serverless functions for automated object detection, tracking, and skeleton annotation. It supports co

FileCentipede is a comprehensive file management and transfer application designed to handle diverse network protocols and data operations. It functions as a multi-protocol download manager, a full-featured BitTorrent client, and a remote filesystem manager, providing a unified interface for moving and organizing data across local and remote environments. The application distinguishes itself through deep browser integration, which allows for the direct capture of media streams, video, and bulk download links from web pages. It also includes a modular utility suite that enables users to perfor

Records is a SQL database client designed for executing raw queries and managing result sets through a simplified interface. It provides a parameterized SQL executor to bind values to placeholders, ensuring safe data handling and preventing injection attacks, alongside a database transaction manager for grouping operations into atomic units. The project includes a dedicated command-line interface for running database statements and exporting query results directly to local files. This tooling allows for the conversion of SQL result sets into multiple serialization formats, including CSV, JSON

HomeBox is a self-hosted home inventory manager designed for tracking physical belongings and household assets. It functions as a digital catalog for creating structured databases of objects, including records of locations, categories, and purchase history. The system distinguishes itself through the use of QR code generation to link physical objects to digital records and the support of hierarchical location mapping to track assets across nested environments. It further enables automation via a REST API and centralizes access management through OpenID Connect integration for user authenticat

StreamPark 是一个集中式管理平台，旨在协调分布式流处理和批处理应用程序的部署、监控和操作生命周期。它作为一个数据流水线控制平面和编排器，专门为 Apache Flink 和 Hadoop YARN 环境提供管理功能。 该平台通过任务部署的低代码方法和支持多种处理运行时的多引擎执行适配器脱颖而出。它通过将流式 SQL 分析与处理版本控制、二进制上传和基于保存点 (savepoint) 的状态恢复的资源驱动部署流水线相结合，促进了实时数据流水线管理。 该系统涵盖了广泛的功能，包括分布式作业编排、通过预构建连接器进行的实时数据集成，以及通过 LDAP 或 SSO 进行的身份集成。它还提供用于秒级应用程序监控和自动化操作故障通知的可观测性工具。

GAM is a command-line tool for administering Google Workspace and Cloud Identity. It translates command-line arguments into structured API calls, enabling administrators to manage users, groups, organizational units, and domain settings across a Google Workspace environment. The tool handles authentication through OAuth2 flows, service accounts, and workload identity federation, and supports multi-tenant configurations for managing multiple domains or cloud projects from a single installation. GAM distinguishes itself through its batch processing and automation capabilities. It can process la

imapsync is an IMAP mailbox synchronization tool and data migration utility designed to copy and synchronize email messages and folder structures between two IMAP servers. It functions as a migration manager for transferring bulk email accounts between different hosting providers, preserving folder hierarchies and message metadata. The tool is distinguished by its ability to automate the transfer of multiple mailboxes sequentially from delimited lists using administrative credentials or user-specific authentication. It supports advanced authentication methods including OAuth2 and XOAUTH2, and