A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
apache/incubator-gobblin की मुख्य विशेषताएं हैं: Data Ingestion Pipelines।
apache/incubator-gobblin के ओपन-सोर्स विकल्पों में शामिल हैं: apache/iggy — Iggy is a distributed message streaming platform and multi-protocol message broker that functions as a persistent… bahador-r/db2lake. apache/pulsar — Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It… awslabs/aws-data-wrangler — This project is an AWS pandas integration library and data pipeline framework designed to simplify the movement and… aklivity/zilla — 🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka®… airbnb/kafkat — KafkaT-ool.
Iggy is a distributed message streaming platform and multi-protocol message broker that functions as a persistent distributed log store. It provides infrastructure for publishing and consuming binary messages using an append-only log, ensuring high availability and data consistency across nodes through Viewstamped Replication. The platform is distinguished by its specialized LLM streaming infrastructure, which uses a server protocol to connect large language models to streaming data and system controls. This includes standardized protocols for context management and data bridging via HTTP or
🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.
Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It functions as a geo-replicated data streamer and a multi-tenant event streaming platform, providing a serverless stream processing engine and a tiered storage messaging broker. The system distinguishes itself by separating serving layers from storage layers to allow independent scaling of compute and data retention. It features native geo-replication to synchronize messages across different geographical regions and employs a multi-layered tenant isolation model using authentica
