Awesome GitHub RepositoriesData Ingestion

Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It functions as a geo-replicated data streamer and a multi-tenant event streaming platform, providing a serverless stream processing engine and a tiered storage messaging broker. The system distinguishes itself by separating serving layers from storage layers to allow independent scaling of compute and data retention. It features native geo-replication to synchronize messages across different geographical regions and employs a multi-layered tenant isolation model using authentica

Rudder Server عبارة عن منصة بيانات عملاء وخط أنابيب توجيه أحداث مصمم لجمع وتحويل وتوجيه بيانات أحداث العملاء من مصادر مختلفة إلى مستودعات البيانات وأدوات الأعمال. يعمل كمحلل هوية عملاء، يربط المعرفات من مصادر متعددة لبناء رسم بياني موحد للهوية وملفات تعريف سلوكية شاملة للعملاء. يتميز النظام بقدرات ETL العكسية، التي تدفع شرائح العملاء والجماهير المعالجة من مستودعات البيانات مرة أخرى إلى تطبيقات الطرف الثالث التشغيلية. كما يوفر مستوى بيانات حاوية لنشر Kubernetes، مما يتيح إدارة البنية التحتية للبيانات ككود. تغطي المنصة مجموعة واسعة من قدرات إدارة البيانات، بما في ذلك تحويل الأحداث في الوقت الفعلي، والتحقق من المخطط عبر كتالوجات البيانات، وحوكمة الخصوصية. تشمل هذه أدوات لإدارة موافقة المستخدم، وفرض إقامة البيانات داخل مناطق جغرافية محددة، وإخفاء معلومات التعريف الشخصية أثناء النقل. تتم إدارة تثبيت ونشر مكونات مستوى البيانات باستخدام مخططات Helm.

Scribe is a distributed log aggregation system designed to collect and route real-time log data from numerous servers to centralized storage or analysis tools. It functions as a log data pipeline and scalable collector that gathers streaming data and writes it to local disks or remote endpoints. The system employs a log routing server model that organizes incoming streams into specific buckets based on predefined configuration mappings. It supports multi-hop log forwarding, allowing data to be routed through a chain of intermediate servers to centralize logs from diverse network segments. Re

ingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations. The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets. The system pr

DEPRECATED: Data collection and processing made easy.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Secor is a service implementing Kafka log persistence

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Netflix's distributed Data Pipeline

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

A probabilistic data structure service and storage

🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.

Hadoop log aggregator and dashboard

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

DocId set compression and set operation library