Awesome GitHub RepositoriesData Ingestion

Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It functions as a geo-replicated data streamer and a multi-tenant event streaming platform, providing a serverless stream processing engine and a tiered storage messaging broker. The system distinguishes itself by separating serving layers from storage layers to allow independent scaling of compute and data retention. It features native geo-replication to synchronize messages across different geographical regions and employs a multi-layered tenant isolation model using authentica

Rudder Server is a customer data platform and event routing pipeline designed to collect, transform, and route customer event data from various sources to data warehouses and business tools. It functions as a customer identity resolver, linking identifiers from multiple sources to build a unified identity graph and comprehensive behavioral customer profiles. The system differentiates itself through reverse ETL capabilities, which push processed customer segments and audiences from data warehouses back into operational third-party applications. It also provides a containerized data plane for K

Scribe este un sistem distribuit de agregare a log-urilor, conceput pentru a colecta și ruta date de log în timp real de la numeroase servere către stocare centralizată sau instrumente de analiză. Funcționează ca un pipeline de date de log și un colector scalabil care adună datele în flux și le scrie pe discuri locale sau endpoint-uri la distanță. Sistemul folosește un model de server de rutare a log-urilor care organizează fluxurile primite în „buckets” specifice, bazate pe mapări de configurare predefinite. Suportă redirecționarea log-urilor prin mai multe hop-uri, permițând rutarea datelor printr-un lanț de servere intermediare pentru a centraliza log-urile din diverse segmente de rețea. Fiabilitatea și observabilitatea sunt gestionate prin buffering pe disc local, care stochează mesajele de ieșire în timpul defecțiunilor de rețea pentru a preveni pierderea datelor, și monitorizarea sănătății bazată pe contoare pentru a urmări numărul intern de mesaje și succesul livrării. Arhitectura include, de asemenea, streaming asincron de mesaje și rutare centralizată a log-urilor pentru a gestiona fluxurile de date cu throughput ridicat în rețea.

ingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations. The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets. The system pr

DEPRECATED: Data collection and processing made easy.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Secor is a service implementing Kafka log persistence

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Netflix's distributed Data Pipeline

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

A probabilistic data structure service and storage

🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.

Hadoop log aggregator and dashboard

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

DocId set compression and set operation library