benthosdev/benthos
Benthos
Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing.
The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations without custom code.
The platform provides an observability-driven data plane with integrated telemetry, performance metrics, and message flow tracing. Reliability is managed through a transaction model that ensures at-least-once delivery guarantees to prevent data loss during system crashes.
The engine is extensible through a plugin architecture that supports loading external binary modules to add new source and sink connectors.
Features
- Data Ingestion and Integration - Moves and integrates data across different platforms by consuming from sources, transforming, and delivering to destinations.
- Data Integration Pipelines - The product enables moving data between various sources and sinks using a declarative pipeline configuration defined within a single file.
- Stream Processing Engines - Provides a declarative engine for routing, transforming, and connecting high-velocity data streams between diverse sources and sinks.
- Data Pipelines and ETL - Functions as a framework for building data transformation and integration pipelines with built-in filtering.
- Change Data Capture - Streams real-time database modifications as discrete events for downstream processing.
- Declarative Workflow Definitions - Simplifies stream management by defining data movement and transformation via a single configuration file.
- Declarative Pipeline Construction - Defines data workflows as static graphs via a single configuration file that is optimized before execution.
- Data Transformation Languages - Uses a specialized declarative language to perform complex structural modifications on data payloads.
- Real-Time Data Streaming - Processes and delivers live data streams in real-time using a specialized language for enrichment.
- Stream Routing - Provides a modular system for routing data streams between diverse inputs and outputs.
- Stream Transformations - Performs real-time processing of data streams using a specialized language for filtering, mapping, and enrichment.
- Message Delivery Guarantees - Implements a transaction model with offsets and acknowledgments to ensure at-least-once delivery of messages.
- At-Least-Once Delivery Guarantees - Ensures data is not lost during system crashes using a transaction model that guarantees at-least-once delivery.
- Event Middleware - Provides the infrastructure for decoupling services by consuming and delivering messages via a transaction model.
- Change Data Capture Tools - Acts as a tool to monitor and stream database modifications to external systems in real-time.
- Extensible Plugin Architectures - Provides a modular architecture for adding custom functionality via external binary plugins.
- Plugin-Based Architectures - Supports loading external binary modules at runtime to extend the engine with new source and sink connectors.
- End-to-End Message Tracing - Generates telemetry traces to visualize the end-to-end movement of data through the processing pipeline.
- Pipeline Health Monitors - Exposes health probes and emits performance metrics to external monitoring backends for operational oversight of data pipelines.
- Internal Telemetry Monitors - Integrates health probes and performance metrics to expose the internal operational state of the processing pipeline.
- Metric and Performance Monitors - Monitors pipeline health and performance through integrated telemetry and high-frequency numerical metrics.
- Observability Data Planes - Emits performance metrics and telemetry traces to monitor the health of streaming data flows in real-time.
- Data Integration - Message streaming bridge for various protocols.
