robinhood/faust
Faust
Faust is a Python library for building distributed stream processing applications that integrate with Kafka. It functions as an asynchronous stream processor designed to handle high-throughput event streams and real-time data analysis using asynchronous functions.
The system operates as a distributed stream processor and state store, utilizing sharding and partitioned topics to scale processing workloads horizontally across multiple worker nodes. It maintains state through a replicated key-value storage system backed by local databases to ensure high availability and fast recovery.
The framework provides capabilities for windowed data aggregation to track trends over time, joining multiple data streams, and defining serialized data models via type annotations. It also includes tools for monitoring system performance and visually inspecting stored key-value pairs through an HTTP interface.
Features
- Real-Time Data Streaming - Provides a platform for processing and delivering event-driven data streams in real-time.
- Distributed Key-Value Stores - Maintains a consistent and replicated key-value state store across multiple processing nodes.
- Streaming Processors - Operates as a distributed processing engine that handles continuous event sequences across multiple worker nodes.
- Stream Processing - Provides a framework for the continuous ingestion, transformation, and analysis of high-velocity Kafka data streams.
- Distributed Sharding Architectures - Distributes stream workloads across multiple worker nodes by routing events with the same key to the same processor.
- Stream Processing Libraries - Provides a comprehensive Python library for building distributed stream processing applications integrated with Kafka.
- Local State Stores - Maintains a persistent key-value database on each node to store large amounts of state locally.
- Real-Time Event Processing - Executes asynchronous computation and application logic for every single event received from a messaging queue.
- Distributed State Management - Maintains consistent application state and replicated key-value data across distributed worker nodes for high availability.
- Asynchronous Event Loops - Uses a non-blocking event loop to handle high-throughput data streams and concurrent network requests.
- Event-Driven Frameworks - Provides a framework for building high-concurrency applications that transform data from messaging topics using asynchronous functions.
- Asynchronous Stream Processors - Uses an asynchronous event loop and non-blocking functions to handle high-throughput event streams and real-time analysis.
- Data Joins - Combines multiple real-time streams or tables using join operations to correlate event data.
- Data Serialization - Implements mechanisms for encoding, parsing, and serializing data structures using type annotations.
- Data Serialization Formats - Defines structured data models and serialization formats to ensure consistent formatting for stream keys and values.
- Time-Window Aggregations - Tracks data summaries over sliding or tumbling time intervals to analyze trends within continuous streams.
- Horizontal Scaling - Provides mechanisms to increase processing capacity by distributing partitioned topics across multiple worker nodes.
- Horizontal Scaling Strategies - Implements architectural methods to expand system throughput by sharding partitioned topics across multiple worker instances.
- Log Replication Strategies - Ensures high availability by treating state updates as a transaction log that can be replicated and recovered.
- Serialization Models - Uses Python type hints to define data models for automatic serialization and deserialization of stream messages.
- Streaming Engines - Python stream processing library inspired by Kafka Streams.
