Awesome GitHub RepositoriesData Integration Pipelines

DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data

Azure Docs is the official technical documentation repository for Microsoft Azure, the cloud computing platform. It provides comprehensive guidance on the full spectrum of Azure services, covering everything from core infrastructure components like virtual machines, Kubernetes clusters, and serverless computing to platform services for AI, machine learning, data analytics, and storage. The documentation details how to provision, manage, and govern cloud resources at scale, including policy enforcement, identity management, and cost optimization. The documentation distinguishes Azure through i

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

This project is a streaming data integration framework that captures real-time database changes and synchronizes them with downstream systems. It operates as a distributed streaming ETL and database synchronizer, reading database logs and snapshots to propagate row-level modifications to target sinks. The system supports declarative data integration, allowing users to define source-to-sink data flows using SQL or YAML configurations. It distinguishes itself by automating schema evolution to maintain synchronization when source structures change and ensuring exactly-once delivery and processin

Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data pipelines. At its core, it provides a web-based visual dataflow designer where users build directed graphs of processors to route, transform, and mediate data movement between any source and destination without writing custom code. The system records fine-grained data provenance for every data item from ingestion to delivery, supporting audit, debugging, and replay of data lineage. The platform distinguishes itself through a zero-master cluster architecture that distributes proc

Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data

Briefer هي منصة دفاتر بيانات تفاعلية وأداة لوحة تحكم ذكاء الأعمال تُستخدم لتحليل البيانات والتقارير التعاونية. توفر بيئة معتمدة على الحاويات لبناء تقارير تجمع بين SQL، وPython، وMarkdown مع تصورات أصلية. تتميز المنصة بمساعد كود مدمج يستخدم نماذج لغة كبيرة لتوليد مقتطفات SQL و Python من مطالبات اللغة الطبيعية. وهي مصممة كتطبيق بيانات Kubernetes، حيث يتم النشر عبر Helm charts لإدارة بيئات الحوسبة المعزولة وضمان موارد منفصلة لكل صفحة من خلال العزل القائم على الـ pod. يغطي النظام نطاقاً واسعاً من القدرات بما في ذلك اتصال قاعدة البيانات الخارجية، والتحرير المشترك في الوقت الفعلي، وتسليم التقارير المؤتمتة عبر الجدولة. تتكامل مع OpenID Connect لتوفير الهوية وتوفر تحكماً في الوصول قائماً على الأدوار، وإدارة آمنة للاعتمادات، وتخزين مؤقت للاستعلام قائم على النتائج. يتم نشر التطبيق وتوسيع نطاقه عبر مجموعات Kubernetes باستخدام Helm charts مدارة.

Console هي أداة إدارة تعتمد على الويب لمراقبة وإدارة مجموعات Kafka. توفر واجهة رسومية لإدارة المواضيع (topics)، ومجموعات المستهلكين، وتدفقات البيانات في الوقت الفعلي. يتضمن المشروع مديرين مخصصين لـ Kafka Connect و Schema Registry، مما يسمح بنشر موصلات البيانات وحوكمة مخططات البيانات. يتميز بلوحة تحكم للمجموعة لتتبع صحة الوسطاء (brokers)، واستخدام التخزين، وحالة الرفوف، إلى جانب واجهة للتحكم في الوصول القائم على الأدوار وفحص سجلات التدقيق. تغطي المنصة تصفح البيانات وتصفيتها، وتنسيق مجموعات المستهلكين، وبناء خطوط أنابيب البيانات. كما تدعم مراقبة مهام تحويل البيانات وإدارة أذونات المستخدم عبر محرك سياسات مركزي.

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind