Awesome GitHub RepositoriesStream Ingestion

Delta is a lakehouse table format that brings ACID transactions and data warehouse consistency to large scale data lakes on cloud object storage. It serves as an ACID transaction manager, coordinating atomic commits and serializable isolation for concurrent reads and writes across distributed compute engines. The project provides a multi-engine interoperability layer that uses format translation to allow diverse SQL engines and processing frameworks to read and write the same tables. It functions as a data versioning system, utilizing a transaction log to enable time travel, historical snapsh

Moto is a cloud service mockery framework and API mock server that simulates AWS infrastructure locally. It allows developers to test cloud-dependent code and verify infrastructure-as-code templates without deploying real resources or incurring costs. The project functions as an SDK interceptor that can patch existing service clients to redirect requests to a local mock environment. It can also be run as a standalone HTTP server, enabling any programming language to interact with the simulated endpoints. The framework covers a vast array of simulated capabilities, including data storage, com

Feast is a machine learning feature store and MLOps data infrastructure layer. It provides a centralized system for managing and serving features across offline training and online production environments, utilizing an online feature serving layer for low-latency retrieval. The project centers on a feature registry that acts as a central catalog for defining, governing, and discovering feature services. It employs a unified data access layer to decouple feature retrieval from physical storage and includes a point-in-time data generator to create historically accurate training datasets that pr

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal indexing to data lakes. It provides atomic commits with snapshot isolation, rollback, and optimistic concurrency control for reliable data lake operations, while supporting upserts, record-level updates, and deletions in large analytical datasets. The project distinguishes itself through a timeline-based architecture that coordinates all write operations, enabling features like time-travel querying, incremental change streaming, and multi-modal query views that include snapshot, i

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

dlt este un instrument de ingestie a datelor Python și un framework de pipeline ETL conceput pentru a prelua date din surse diverse și a le persista în destinații structurate. Funcționează ca un motor de inferență a schemei care detectează automat tipurile de date și aplatizează structurile JSON imbricate în tabele relaționale, mutând datele din surse către lakehouse-uri, depozite de date sau baze de date vectoriale. Proiectul se distinge prin generarea de pipeline-uri bazată pe AI, utilizând modele lingvistice mari pentru a crea codul de extracție și conectorii pentru API-urile REST. De asemenea, suportă stocarea vectorială multimodală și popularea specializată a bazelor de date vectoriale pentru a susține aplicațiile AI și machine learning. Framework-ul acoperă o gamă largă de capabilități, inclusiv evoluția automată a schemei, încărcarea incrementală a datelor prin urmărirea stării și validarea calității datelor prin aplicarea contractelor de date. Oferă instrumente pentru normalizarea datelor relaționale, transformări pre- și post-încărcare și o varietate de adaptoare de destinație pentru baze de date SQL și stocare de obiecte în cloud. Observabilitatea este gestionată prin dashboard-uri de execuție a pipeline-ului, urmărirea lineage-ului coloanelor și verificarea versiunii schemei folosind hash-uri bazate pe conținut.

OpenPanel is a self-hosted product analytics platform designed for tracking user behavior and visualizing product metrics on private infrastructure. It provides a comprehensive system for collecting events across web, mobile, and server environments while ensuring complete ownership of data. The platform distinguishes itself through a privacy-first approach, utilizing cookieless event tracking and regional data residency to simplify regulatory compliance. It integrates large language models via the Model Context Protocol, enabling users to query behavioral data and analyze trends using natura

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

IntelOwl este o platformă de threat intelligence și un motor de orchestrare a securității conceput pentru a agrega, analiza și îmbogăți observabilele de securitate. Funcționează ca un instrument de investigare a incidentelor de securitate și un agregator de threat intelligence, colectând date despre fișiere, domenii și adrese IP din diverse surse interne și externe. Sistemul se diferențiază prin automatizarea fluxului de lucru bazată pe playbook-uri, permițând utilizatorilor să definească secvențe reutilizabile de sarcini de analiză care declanșează joburi ulterioare pe baza rezultatelor anterioare. Unifică datele de securitate disparate într-o schemă comună și utilizează controale de acces la nivel de protocol pentru a restricționa execuția analizatoarelor în funcție de sensibilitatea datelor. Platforma acoperă o gamă largă de capabilități, inclusiv ingestia de indicatori bazată pe evenimente, fluxuri de lucru automatizate pentru centrele de operațiuni de securitate (SOC) și îmbogățirea inteligenței. Oferă instrumente pentru organizarea investigațiilor și vizualizarea rezultatelor analizei prin dashboard-uri pentru a corela descoperirile. Proiectul este implementat în Python.

ChatLab is a self-hosted chat database and data pipeline designed to normalize, store, and analyze large-scale social conversation histories. It functions as an analytics platform that uses large language models to extract patterns and insights from messaging data imported from multiple platforms. The system distinguishes itself through an AI-powered analysis engine that utilizes vector-based history analysis and agent-based function calling to summarize conversation trends. It further identifies behavioral patterns by generating visual analytics, including heatmaps, word clouds, and activity

Uptrace is an OpenTelemetry-based observability platform designed to collect, store, and analyze distributed traces, metrics, and logs. It functions as a centralized logging backend, a distributed tracing system, and a metrics engine to monitor application performance and system health. The platform is distinguished by AI-powered operational capabilities, allowing users to query telemetry data and manage monitoring dashboards using natural language. It specifically includes specialized monitoring for generative AI pipelines, tracking token usage and response quality for LLM interactions and r

Jackson-databind is a Java serialization framework and JSON data binding library used to convert Java objects to JSON and vice versa. It functions as a JSON streaming API that generates and parses content as a sequence of tokens, and as a hierarchical data tree mapper that reads data into a node structure for dynamic modification without predefined classes. The project provides a structured mapping process to bind data to objects, allowing for the transformation of complex Java objects into data formats and the reconstruction of objects from those formats. It supports custom data format mappi