Open-source alternatives to Scio

This repository is a collection of reference implementations and distributed data processing algorithms implemented in Java and Scala for cluster computing frameworks. It provides computational recipes for solving complex data processing problems, including large-scale dataset joins, aggregations, and word count tasks. The implementations cover both MapReduce paradigms and Apache Spark integrations, enabling programmatic job submission and execution across distributed node infrastructures. The collection includes specialized utilities for statistical analysis and text processing, such as data

Apache Spark is a unified distributed data processing engine designed for large-scale data analysis and computation graphs. It functions as a distributed machine learning framework, a graph processing system, a real-time stream processor, and a SQL analytics engine. The system enables the execution of distributed SQL querying, large-scale graph analysis, and real-time stream analytics across clusters of machines. It also provides a scalable environment for implementing machine learning algorithms and predictive model development on massive datasets. The engine incorporates relational query e

A curated list of awesome Apache Spark packages and resources.

This project is a community-maintained, open-access directory of high-quality public datasets. It serves as a centralized reference point for researchers, developers, and data scientists to locate reliable information sources across a wide spectrum of industries and scientific fields. By providing a structured index, the repository facilitates the discovery of data necessary for exploratory analysis, machine learning model training, and the development of data-intensive applications. The directory distinguishes itself through a lightweight, platform-agnostic approach to resource indexing that

A curated list of awesome network analysis resources.

A schema-aware Scala library for data transformation

Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

psq - Cloud Pub/Sub Task Queue for Python.

Elasticsearch权威指南中文版

a curated list of awesome streaming frameworks, applications, etc

Scala library for accessing various file, batch systems, job schedulers and grid middlewares.

This project is a curated directory of software, frameworks, and educational resources designed for building, scaling, and maintaining distributed data processing and storage architectures. It serves as a comprehensive index for the distributed computing ecosystem, helping users identify the appropriate tools for managing large-scale information systems. The repository functions as a central hub for data engineering, offering categorized access to technologies that support batch and stream processing, machine learning, and interactive querying. By organizing these resources, it assists in the

A collection of awesome resources for Splunk

The Heroic Time Series Database

spark-bigquery

Low-code tool for automating actions on real time data | Stream processing for the users.

Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.

A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources

A curated list of awesome Qlik extensions and resources for Qlik Sense and QlikView

Apache Flink is a distributed processing engine designed for both high-throughput, low-latency data streams and finite batch workloads. It functions as a stateful stream processor and a SQL stream processing engine, providing a unified runtime to execute relational queries and event-based transformations. The system is distinguished by its ability to manage persistent operator state to ensure exactly-once processing guarantees and consistency during failures. It features specialized capabilities for complex event processing to detect temporal patterns and handles out-of-order events using eve

Kafka is a distributed event streaming platform designed for capturing, storing, and processing real-time data streams across interconnected nodes. It functions as a distributed commit log, providing a fault-tolerant storage mechanism that records state changes sequentially to ensure data consistency and durability across distributed environments. The platform distinguishes itself through a partitioned commit log architecture that enables horizontal scaling and parallel processing of data streams. It integrates a stream processing engine for continuous transformations and aggregations, while