3 repositorios
Awesome GitHub RepositoriesLarge Scale Data Integration Frameworks
Systems designed to move massive volumes of structured and unstructured data between diverse databases and cloud storage.
Distinct from Large-Scale Data Computation: Focuses on the integration and movement of diverse data at scale, rather than just computation or storage management.
Explore 3 awesome GitHub repositories matching data & databases · Large Scale Data Integration Frameworks. Refine with filters or upvote what's useful.
Awesome Large Scale Data Integration Frameworks GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- alibaba/datax
alibaba/DataX
17,241Ver en GitHubDataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Functions as a distributed framework for synchronizing massive volumes of data between heterogeneous sources and destinations.
Java - apache/seatunnel
apache/seatunnel
9,427Ver en GitHubSeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding
Moves massive volumes of structured and unstructured data between diverse databases, cloud storage, and messaging systems.
Javaapachebatchcdc - dotnetcore/dotnetspider
dotnetcore/DotnetSpider
4,137Ver en GitHubDotnetSpider es un framework de rastreo web para .NET y una herramienta de extracción de datos en C# diseñada para el descubrimiento automatizado de páginas web y la recuperación de datos estructurados de Internet a gran escala. Funciona como una librería de web scraping de alto nivel para recopilar información de varios sitios web. El framework proporciona capacidades para el rastreo web automatizado y el scraping de datos a gran escala. Permite la extracción de contenido web para facilitar la creación de bases de datos locales o el análisis de información en línea mediante la automatización web programática dentro del ecosistema .NET. El sistema utiliza un modelo de procesamiento de datos basado en tuberías con manejo de solicitudes asíncronas y ejecución de trabajadores concurrentes. Cuenta con un programador basado en colas de tareas, proveedores de almacenamiento modulares y una implementación orientada a interfaces para lógica de scraping personalizada.
Simplifies the collection of large datasets by extracting specific data points from web pages through a structured process.
C#crawlercross-platformcsharp
