benthosdev/benthos
Benthos
Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing.
The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations without custom code.
The platform provides an observability-driven data plane with integrated telemetry, performance metrics, and message flow tracing. Reliability is managed through a transaction model that ensures at-least-once delivery guarantees to prevent data loss during system crashes.
The engine is extensible through a plugin architecture that supports loading external binary modules to add new source and sink connectors.
Features
- Data Ingestion and Integration - Moves and integrates data across different platforms by consuming from sources, transforming, and delivering to destinations.
- Data Integration Pipelines - The product enables moving data between various sources and sinks using a declarative pipeline configuration defined within a single file.
- Stream Processing Engines - Provides a declarative engine for routing, transforming, and connecting high-velocity data streams between diverse sources and sinks.
- Data Pipelines and ETL - Functions as a framework for building data transformation and integration pipelines with built-in filtering.
- Change Data Capture - Streams real-time database modifications as discrete events for downstream processing.
- Declarative Workflow Definitions - Simplifies stream management by defining data movement and transformation via a single configuration file.
- Declarative Pipeline Construction - Defines data workflows as static graphs via a single configuration file that is optimized before execution.
- Data Transformation Languages - Uses a specialized declarative language to perform complex structural modifications on data payloads.
- Real-Time Data Streaming - Processes and delivers live data streams in real-time using a specialized language for enrichment.
- Stream Routing - Provides a modular system for routing data streams between diverse inputs and outputs.
- Stream Transformations - Performs real-time processing of data streams using a specialized language for filtering, mapping, and enrichment.
- Message Delivery Guarantees - Implements a transaction model with offsets and acknowledgments to ensure at-least-once delivery of messages.
- At-Least-Once Delivery Guarantees - Ensures data is not lost during system crashes using a transaction model that guarantees at-least-once delivery.
- Event Middleware - Provides the infrastructure for decoupling services by consuming and delivering messages via a transaction model.
- Change Data Capture Tools - Acts as a tool to monitor and stream database modifications to external systems in real-time.
- Extensible Plugin Architectures - Provides a modular architecture for adding custom functionality via external binary plugins.
- Plugin-Based Architectures - Supports loading external binary modules at runtime to extend the engine with new source and sink connectors.
- End-to-End Message Tracing - Generates telemetry traces to visualize the end-to-end movement of data through the processing pipeline.
- Pipeline Health Monitors - Exposes health probes and emits performance metrics to external monitoring backends for operational oversight of data pipelines.
- Internal Telemetry Monitors - Integrates health probes and performance metrics to expose the internal operational state of the processing pipeline.
- Metric and Performance Monitors - Monitors pipeline health and performance through integrated telemetry and high-frequency numerical metrics.
- Observability Data Planes - Emits performance metrics and telemetry traces to monitor the health of streaming data flows in real-time.
- Data Integration - Message streaming bridge for various protocols.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
