Awesome GitHub RepositoriesStream Processing Pipeline Deployments

Storm is a distributed stream processing framework designed to execute unbounded computations across a cluster to process real-time data streams. It functions as a data pipeline orchestrator that allows users to define and deploy declarative data flow graphs connecting streaming sources to processing components. The system operates as a multi-tenant distributed compute engine that isolates workloads and limits resource usage across shared clusters using dedicated pools and access control. It is also a secure distributed processing engine that employs encrypted node communication and SSL-secur

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

StreamPark ist eine zentralisierte Managementplattform, die darauf ausgelegt ist, das Deployment, Monitoring und den operativen Lebenszyklus verteilter Stream-Processing- und Batch-Anwendungen zu koordinieren. Sie fungiert als Control-Plane und Orchestrator für Datenpipelines und bietet spezifisch Managementfunktionen für Apache Flink- und Hadoop YARN-Umgebungen. Die Plattform zeichnet sich durch einen Low-Code-Ansatz für das Task-Deployment und einen Multi-Engine-Execution-Adapter aus, der diverse Verarbeitungs-Runtimes unterstützt. Sie erleichtert das Echtzeit-Datenpipeline-Management durch die Kombination von Streaming-SQL-Analytics mit einer ressourcenbasierten Deployment-Pipeline, die Versionierung, Binär-Uploads und Savepoint-basierte Zustands-Wiederherstellung handhabt. Das System deckt ein breites Spektrum an Funktionen ab, einschließlich verteilter Job-Orchestrierung, Echtzeit-Datenintegration über vorgefertigte Connectors und Identitätsintegration via LDAP oder SSO. Es bietet zudem Observability-Tools für sekundengenaue Anwendungsüberwachung und automatisierte operative Fehlerbenachrichtigungen.

Dinky is a real-time data platform for developing, deploying, and operating streaming applications based on Apache Flink. It functions as a SQL streaming IDE and a real-time data pipeline orchestrator, providing a web-based environment for writing and verifying queries with integrated logic plan visualization and lineage tracking. The platform acts as a distributed cluster manager, allowing the registration, monitoring, and administration of multiple processing clusters from a centralized interface. It also serves as a change data capture integration tool, synchronizing real-time database cha