Awesome GitHub RepositoriesData Integration Pipelines

DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data

Azure Docs is the official technical documentation repository for Microsoft Azure, the cloud computing platform. It provides comprehensive guidance on the full spectrum of Azure services, covering everything from core infrastructure components like virtual machines, Kubernetes clusters, and serverless computing to platform services for AI, machine learning, data analytics, and storage. The documentation details how to provision, manage, and govern cloud resources at scale, including policy enforcement, identity management, and cost optimization. The documentation distinguishes Azure through i

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

This project is a streaming data integration framework that captures real-time database changes and synchronizes them with downstream systems. It operates as a distributed streaming ETL and database synchronizer, reading database logs and snapshots to propagate row-level modifications to target sinks. The system supports declarative data integration, allowing users to define source-to-sink data flows using SQL or YAML configurations. It distinguishes itself by automating schema evolution to maintain synchronization when source structures change and ensuring exactly-once delivery and processin

Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data pipelines. At its core, it provides a web-based visual dataflow designer where users build directed graphs of processors to route, transform, and mediate data movement between any source and destination without writing custom code. The system records fine-grained data provenance for every data item from ingestion to delivery, supporting audit, debugging, and replay of data lineage. The platform distinguishes itself through a zero-master cluster architecture that distributes proc

Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data

Briefer ist eine interaktive Daten-Notebook-Plattform und ein Business-Intelligence-Dashboard-Tool, das für kollaborative Datenanalyse und Berichterstattung verwendet wird. Es bietet eine containerisierte Umgebung zum Erstellen von Berichten, die SQL, Python und Markdown mit nativen Visualisierungen kombinieren. Die Plattform verfügt über einen integrierten Code-Assistenten, der Large Language Models verwendet, um SQL- und Python-Snippets aus natürlichsprachlichen Prompts zu generieren. Sie ist als Kubernetes-Datenanwendung konzipiert und wird über Helm-Charts bereitgestellt, um isolierte Rechenumgebungen zu verwalten und separate Ressourcen pro Seite durch Pod-basierte Isolierung sicherzustellen. Das System deckt ein breites Spektrum an Funktionen ab, einschließlich externer Datenbankkonnektivität, Echtzeit-Co-Editing und automatisierter Berichtszustellung via Scheduling. Es integriert sich mit OpenID Connect für die Identitätsbereitstellung und bietet rollenbasierte Zugriffskontrolle, sicheres Credential-Management und ergebnisbasiertes Query-Caching. Die Anwendung wird über Kubernetes-Cluster mittels verwalteter Helm-Charts bereitgestellt und skaliert.

Console ist ein webbasiertes Management-Tool zur Überwachung und Administration von Kafka-Clustern. Es bietet eine grafische Oberfläche zur Verwaltung von Topics, Consumer-Groups und Echtzeit-Datenströmen. Das Projekt umfasst dedizierte Manager für Kafka Connect und die Schema Registry, was das Deployment von Daten-Connectoren und die Governance von Datenschemata ermöglicht. Es bietet ein Cluster-Dashboard zur Verfolgung von Broker-Gesundheit, Speichernutzung und Rack-Status sowie eine Schnittstelle für rollenbasierte Zugriffskontrolle und die Prüfung von Audit-Logs. Die Plattform deckt das Durchsuchen und Filtern von Daten, die Koordination von Consumer-Groups und den Aufbau von Datenpipelines ab. Zudem unterstützt sie die Überwachung von Datentransformationsaufgaben und die Verwaltung von Nutzerberechtigungen über eine zentralisierte Policy-Engine.

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind