ArroyoSystems/arroyo

Features

• Streaming SQL - Translates SQL queries with streaming extensions into an optimized dataflow plan that runs continuously on unbounded data.
• Ad Hoc Dataset Querying - Runs ad-hoc SELECT queries on streaming data with projections, filtering, and subqueries.
• Change Data Capture Streams - Processes CDC-formatted streams from Debezium with stateful upsert and delete semantics.
• Kafka Connectors - Writes records to Kafka topics with exactly-once or at-least-once guarantees and JSON output.
• Real-Time Data Processors - An open-source system for building fault-tolerant, stateful pipelines that process millions of events per second with subsecond latency.
• JSON-Schema - Arroyo reads and writes structured or unstructured JSON data with configurable schema handling and timestamp or decimal encoding.
• Stream Processing - A high-performance stream processing engine built in Rust, supporting SQL queries and user-defined functions.
• Distributed Stream Execution - Executes streaming pipelines as a distributed DAG of parallel subtasks for high throughput and fault tolerance.
• Event-Time Processing - Handles out-of-order events using watermarks and supports tumbling, sliding, and session windows.
• Time-Window Aggregations - Computes running metrics over tumbling, sliding, and session windows based on event-time progress.
• Low-Latency Stream Output - Emits output with subsecond latency by processing data incrementally using the Dataflow model.
• Streaming State Management - Maintains state across streaming events to enable windowed aggregations, joins, and other stateful computations.
• Real-Time Analytics - Computes windowed aggregations, joins, and top-N rankings on streaming data with event-time semantics.
• Streaming SQL Transformations - Defines continuous data transformations and analytics using SQL on unbounded event streams.
• SQL-Based Pipeline Definitions - Defines streaming data pipelines using SQL as the primary language for transformation and analysis logic.
• Kafka Stream Ingestion - Consumes and produces messages from Kafka topics with exactly-once semantics using SQL queries.
• Streaming Source and Sink Tables - Declares source and sink tables using CREATE TABLE SQL with connector options and WITH clauses.
• Streaming Connector Abstractions - Unifies sources and sinks through a common trait interface, supporting Kafka, Kinesis, files, and more with exactly-once guarantees.
• Watermark-Based Event Tracking - Uses watermarks derived from event timestamps to handle out-of-order data and trigger window computations consistently.
• Checkpoint-Based Recovery - Periodically persists consistent snapshots of operator state to remote storage, enabling exactly-once recovery and rescaling.
• Pipeline Execution Monitors - Starts a streaming computation, shows real-time operator metrics and outputs, and takes periodic state snapshots for recovery.
• State Checkpointing - Periodically saves pipeline state to remote storage for exactly-once fault tolerance and recovery.
• CLI and Web GUI Operation Interfaces - Provides a web-based user interface and REST API to create, configure, and monitor real-time data processing pipelines.
• Orchestrator-Worker Models - Schedules stream processing workers across local processes, dedicated nodes, or Kubernetes pods with configurable resources.
• Visual Dataflow Graph Designers - Executes streaming pipelines as a directed acyclic graph of parallel operators, routing data via forward or shuffle edges across workers.
• State Persistence - Persists pipeline state, schemas, and checkpoint data to SQLite, Postgres, or object stores for fault tolerance.
• Streaming Operator Execution - Runs all operators and user-defined functions as compiled native code, providing high throughput and low latency.
• Data Formats and Parsing - Arroyo supports JSON, Avro, Parquet, and raw string formats for reading and writing data through connectors.
• JSON Array Functions - Provides built-in SQL functions for JSONPath extraction and array transformation on streaming data.
• Change Data Capture - Ingests real-time database change events from MySQL and Postgres via Debezium and presents them as queryable streams.
• Built-in Metadata Columns - Arroyo adds metadata such as partition and offset as columns in source tables for use in SQL queries.
• Avro Connector Serializations - Arroyo reads and writes Avro binary data, supporting Confluent Schema Registry and flexible serialization modes for schema distribution.
• Reusable Stream Connections - Arroyo creates reusable source and sink definitions through the Web UI or SQL that can be shared across queries.
• Named Struct Columns - Supports nested struct columns with named fields and dot-notation access in streaming SQL.
• CDC Sources - Ingests change data capture streams from MySQL and Postgres databases via Debezium.
• Stream Lookup Joins - Enriches streaming data by querying external key-value stores like Redis with lookup joins.
• Database Type Support - Supports array columns as a built-in SQL data type with indexing and unnesting operations.
• Horizontal Scaling - Distributes stream processing across multiple workers to handle high throughput with horizontal scaling.
• Parquet Data Exports - Arroyo writes data in Parquet columnar format to file system sinks for efficient storage in data lakes.
• Protobuf Serialization - Arroyo reads Protocol Buffers binary data with Confluent Schema Registry or custom schema definitions via the Web UI or API.
• Stream Schema Enforcers - Arroyo enforces a schema for source data with support for JSON Schema, Avro, Protobuf, and automatic retrieval from Schema Registry.
• Custom SQL Functions - Defines custom SQL functions in Rust or Python for use in streaming data pipelines.
• Derived Stream Definitions - Defines new data streams from SQL queries with automatic schema inference from the query result.
• Window Functions - Provides SQL window functions for ranking and analytical calculations over partitioned rows in streaming data.
• Pipeline Execution Interfaces - Starts a stream processing pipeline directly from the command line, accepting SQL from standard input or as an argument.
• Stream Processing Pipeline Deployments - Deploys fault-tolerant stream processors on Kubernetes with checkpointing and a web UI for monitoring.
• Timestamp Construction Functions - Provides SQL functions for constructing, converting, extracting, and truncating timestamps in streaming queries.
• Primitive Types - Provides standard primitive SQL types including integers, floats, strings, and timestamps.
• Type Conversion and Casting - Provides built-in SQL functions for casting, null substitution, binary encoding, and struct construction.
• SQL Mathematical Functions - Provides a comprehensive library of arithmetic, trigonometric, logarithmic, and hash functions for SQL queries.
• API Authentication - Supports static API keys and mutual TLS for API access control.
• Event Timestamp Accessors - Provides SQL functions to access event timestamps and pipeline start times for time-based processing.
• Schema Registries - Arroyo fetches and applies JSON and Avro schemas automatically from a centralized registry for both sources and sinks.
• String Manipulation - Offers a comprehensive set of SQL string functions including regex, substring, and transformation operations.
• Synthetic Data Generation - Generates synthetic streaming events at configurable speeds for testing pipeline behavior.
• Stream Join Operators - Joins unbounded streams incrementally, outputting a changelog of inserts, updates, and deletes.
• Streaming Engines - Distributed engine supporting SQL and Rust pipelines.

Star history