Awesome GitHub RepositoriesLarge Scale Data Integration Frameworks

DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

DotnetSpider 是一个 .NET 网络爬虫框架和 C# 数据提取工具，旨在实现大规模的自动化网页发现和互联网结构化数据检索。它作为一个高级 Web 抓取库，用于从各种网站收集信息。 该框架提供了自动化网络爬虫和大规模数据抓取的功能。它支持网页内容提取，以便在 .NET 生态系统内通过程序化 Web 自动化创建本地数据库或分析在线信息。 系统利用基于流水线的数据处理模型，具有异步请求处理和并发工作线程执行功能。它具有基于任务队列的调度器、模块化存储提供程序以及用于自定义抓取逻辑的接口驱动实现。