risingwavelabs/risingwave

Features

• Streaming Data Processing - Executes SQL queries in a continuous loop to analyze and transform data streams in real time.
• Materialized Views - Builds and maintains real-time materialized views of streaming data with support for cascading dependencies.
• Real-Time Data Streaming - Processes continuous data flows using SQL for real-time filtering, enrichment, and aggregation.
• Log-Structured Merge-Trees - Persists system state and data using a log-structured merge-tree backed by cloud object storage.
• Change Data Capture - Identifies and streams row-level database changes in real-time to synchronize external state.
• Continuous Data Imports - Connects to external message brokers and databases to continuously import live data.
• Data Ingestion - Provides hybrid ingestion capabilities to stream events from brokers and webhooks while loading historical data from object storage.
• Schema Evolution - Tracks database schema changes and maintains version history in object storage to ensure consistent data processing.
• Kafka Connectors - Consumes payloads from Kafka topics to integrate event-streaming infrastructure into the processing pipeline.
• Exactly-Once Processing Semantics - Guarantees data consistency and completeness through exactly-once processing semantics, even during node failures.
• Streaming Data Lakehouses - Integrates streaming ingestion with open table formats like Apache Iceberg for unified batch and stream processing.
• Real-Time Data Processors - Ingests and transforms data streams in real-time using SQL for continuous analytics and event handling.
• Windowed Event Aggregations - Groups continuous event flows into time intervals to calculate periodic metrics.
• Downstream System Synchronization - Delivers transformed streaming results to data warehouses, lakes, or brokers to keep downstream systems current.
• Data Sinking - Continuously pushes processed data streams into downstream databases, data lakes, or message queues.
• Multi-Source Ingestion - Provides the ability to ingest real-time event streams from various sources including message brokers, object storage, and webhooks.
• Cursor-Based Change Streaming - Delivers real-time updates from materialized views using a cursor-based subscription mechanism.
• Streaming Platforms - Provides a distributed streaming platform that decouples compute from state using cloud object storage.
• Logical Replication Ingestion - Streams database changes in real-time using embedded engines and logical replication.
• Event Streaming Pipelines - Transforms raw event streams from brokers like Kafka into structured formats for data lakes and warehouses.
• Event-Time Processing - Handles event-time and processing-time attributes to ensure correctness in windowed operations and out-of-order data.
• Time-Window Aggregations - Computes running totals and metrics using various time-windowing strategies based on event-time progress.
• Continuous Data Export - Outputs computation results continuously to external databases, message queues, or object storage.
• Fault Tolerant Storage - Implements fault-tolerant storage with internal checkpoints to ensure reliability for long-running streaming pipelines.
• Lakehouse Ingestion Pipelines - Streams data into open table formats with exactly-once delivery, automated compaction, and snapshot management.
• High-Throughput Data Streaming - Processes hundreds of thousands of records per second with optimized CPU and memory utilization.
• Incremental Computation - Executes incremental aggregations and joins to maintain real-time views of streaming data.
• Integrated Storage Engines - Stores processing state within an integrated database storage layer to eliminate external state backends.
• Cascading - Allows the construction of layered streaming transformations by building materialized views on top of other materialized views.
• Incremental View Refreshes - Updates pre-computed query results in real-time by calculating only the changes as new events arrive.
• Cascading View Pipelines - Builds layers of streaming transformations by creating materialized views as inputs for subsequent views.
• Message Brokers - Ingests real-time event streams from message brokers to enable immediate data freshness for analytics.
• Data Ingestion - Implements mechanisms for importing streaming data from message brokers, object storage, and databases into the system.
• PostgreSQL-Compatible Databases - Uses the PostgreSQL wire protocol to ensure seamless integration with existing SQL tools and drivers.
• PostgreSQL Protocol Compatibility - Implements the PostgreSQL wire protocol to ensure interoperability with existing SQL tools and drivers.
• Streaming State Management - Maintains low-latency state for complex streaming operations including anti-joins and dynamic filtering.
• Advanced SQL Construct Execution - Supports execution of advanced SQL features including window functions and temporal filters across diverse data types.
• Real-Time Analytics - Maintains automatically updating materialized views for instant retrieval of pre-computed answers via SQL.
• Real-time Analytics Platforms - Computes incremental aggregations and joins on live event streams for low-latency analytical querying.
• Real-Time Data Aggregators - Implements real-time incremental aggregations that update pre-computed query results as new data arrives.
• Real-time Data Subscriptions - Provides capabilities to subscribe to live database updates and trigger application logic in real-time.
• Real-Time Event Processing - Computes results continuously as events arrive to enable immediate monitoring, alerting, and fraud detection.
• Real-Time State Maintenance - Update materialized views incrementally as events arrive to allow SQL queries of the current state in the project.
• SQL Query Execution Engines - Executes structured query language commands to ingest, process, and retrieve real-time data.
• Storage-Compute Architectures - Separates processing nodes from remote object stores to allow independent scaling and rapid recovery.
• Log-Structured Merge-Trees - Stores internal processing state in a log-structured merge-tree backed by cloud object storage.
• Object - Stores streaming state and materialized views in object storage to enable independent scaling and rapid recovery.
• Stream Enrichment - Joins streaming events with reference data from databases and automatically updates outputs when reference data changes.
• Streaming SQL Databases - Processes continuous data streams and maintains materialized views using standard SQL.
• Streaming SQL Transformations - Uses SQL engines to execute filtering, aggregation, and restructuring directly against live data streams.
• SQL-Based Pipeline Definitions - Allows the definition of streaming pipelines using standard SQL, eliminating the need for custom application code.
• Streaming State Recovery - Saves incremental operator states and output results to durable remote storage to ensure rapid system recovery.
• Watermark-Based Event Tracking - Handles out-of-order data using watermarks to ensure accurate windowed aggregations in real-time streams.
• Event Ingestion - Pulls real-time data from external message queues for continuous processing and analytical pipelines.
• Storage Decoupling - Employs a decoupled compute and storage architecture, persisting streaming state in cloud object storage for independent scaling.
• Database Change Streaming - Tracks row-level changes from source databases and delivers them as a real-time event stream.
• Change Stream Semantics - Processes changelog streams using append, update, and delete operations to keep materialized views accurate in real time.
• Streaming Data Integration - Integrates live streaming data with retrieval-augmented generation to power AI applications with current documentation.
• Asynchronous Write Buffering - Decouples streaming processes from downstream write speeds using buffers to prevent pipeline blocks.
• Iceberg Table Lifecycle Management - Handles the lifecycle of Iceberg tables, including catalog management and automated compaction.
• Bulk Data Loading - Provides high-performance utilities for importing massive historical datasets and static files from cloud storage.
• Change Materialization - Processes update and delete operations from external databases by materializing the resulting streams into internal tables.
• Columnar Databases - Employs columnar data structures and specialized table formats to accelerate analytical query performance.
• Point-In-Time Snapshots - Retrieves the state of tables and materialized views as they existed at a specific point in the past.
• Streaming Data Cleaning & Enrichment - Provides built-in SQL transformations to continuously clean, join, and enrich incoming data streams.
• Stateful Operator Acceleration - Increases processing speed by preloading state tables into memory and preserving data locality during backfills.
• Parallel Query Execution - Allows controlling the number of CPU cores allocated to execute streaming and batch query fragments in parallel.
• Disk Caching Systems - Caches frequently accessed data and state on local disks to reduce retrieval costs from cloud object storage.
• External System Synchronization - Streams processed data into external data stores and downstream systems via source and sink connectors.
• Third-Party Data Connectors - Streams data to and from third-party databases, cloud warehouses, and message queues.
• Concurrent Analytical Serving - Runs concurrent SQL queries against streaming data using a dedicated serving layer.
• Parallel Historical Backfilling - Loads large historical datasets using parallel range slicing and lock-free snapshotting to minimize production impact.
• Resource Scaling Strategies - Decouples processing nodes from persistent object storage to scale resources based on specific workload needs.
• JSON CDC Ingestion - Captures real-time database changes in JSON format from topics to synchronize downstream systems.
• Low-Latency Analytical Queries - Processes batch queries with high concurrency and low latency to deliver real-time results.
• Medallion Architecture Support - Maintains data through incremental computation and materialized views to bridge real-time streaming with data lakes.
• Local Disk Caching - Caches data on local disks to minimize the frequency and cost of requests to remote object storage.
• Parallel Data Backfilling - Accelerates historical data loading through range slicing and snapshotting without blocking production traffic.
• Direct Result Serving - Provides direct SQL access to streaming computation results without requiring a separate database.
• Real-Time Feature Computation - Provides the ability to join live event streams with historical reference data to create real-time features for monitoring or AI.
• Incremental Insight Generation - Processes raw data streams incrementally to derive real-time insights into trends and health metrics as events occur.
• Resource Allocation - Assigns dedicated nodes for ingestion or batch execution to prevent resource competition in production.
• Real-Time Result Delivery - Pushes computed real-time data to downstream systems such as serving databases and search indexes.
• Serving Layer Interfaces - Queries real-time data directly using a built-in serving layer and standard SQL.
• Open Table Formats - Writes data to open table formats and automates maintenance tasks like compaction and snapshot cleanup.
• Iceberg Table Materializers - Persists transformed streaming data using the open-source Iceberg table format for cross-environment data sharing.
• User Behavior Analysis - Provides tools to analyze user activity patterns and identify high-value or high-risk user segments.
• Iceberg Table Ingestion - Ingests data from tables managed by external systems using the Apache Iceberg format for cross-environment data exchange.
• Web-Based SQL Consoles - Offers a web-based console for executing SQL queries and managing cluster health and operations.
• Workload Isolation - Isolates batch serving workloads on dedicated clusters to prevent resource contention with continuous streams.
• Backpressure Controllers - Signals upstream sources to slow down ingestion when downstream operators reach resource limits.
• Webhook - Receives data directly from webhooks or external APIs without requiring an intermediate message broker.
• Multi-Cluster Management Systems - Provides a centralized control plane for managing the operational administration of multiple distributed clusters.
• Traffic Throttling - Throttles data consumption from upstream sources via backpressure to prevent system memory exhaustion.
• Distributed Consistency Snapshots - Restores the system to a consistent state using periodic global snapshots stored in remote object storage.
• Load-Aware Throttling - Limits the number of records processed per second based on memory usage to prevent exhaustion during view creation.
• Anomaly Detection - Identifies suspicious behavior and operational anomalies in real-time using automated monitoring systems.
• Cluster Monitoring Dashboards - Provides a web interface to visualize execution graphs, back-pressure rates, and catalog objects.
• Operational Task Automation - Offers a user interface to execute metadata snapshots, diagnostic collection, and administrative maintenance tasks.
• Stream Performance Analysis - Measures end-to-end latency and throughput to identify bottlenecks and tune stream processing efficiency.
• System Restoration - Restores system availability after failures by diagnosing root causes and applying resource adjustments.
• Threshold Monitoring - Evaluates incoming data against defined thresholds in real-time to detect anomalies and trigger alerts.
• Database Systems - Cloud-native streaming database for real-time data.
• Stream Processing - Unifies stream processing and low-latency serving.
• Streaming Engines - PostgreSQL-compatible streaming database for event-driven apps.

Star history