Benthos

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing.

The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations without custom code.

The platform provides an observability-driven data plane with integrated telemetry, performance metrics, and message flow tracing. Reliability is managed through a transaction model that ensures at-least-once delivery guarantees to prevent data loss during system crashes.

The engine is extensible through a plugin architecture that supports loading external binary modules to add new source and sink connectors.

Features

Data Ingestion and Integration - Moves and integrates data across different platforms by consuming from sources, transforming, and delivering to destinations.

Data Integration Pipelines - The product enables moving data between various sources and sinks using a declarative pipeline configuration defined within a single file.

Stream Processing Engines - Provides a declarative engine for routing, transforming, and connecting high-velocity data streams between diverse sources and sinks.

Data Pipelines and ETL - Functions as a framework for building data transformation and integration pipelines with built-in filtering.

Change Data Capture - Streams real-time database modifications as discrete events for downstream processing.

Declarative Workflow Definitions - Simplifies stream management by defining data movement and transformation via a single configuration file.

Declarative Pipeline Construction - Defines data workflows as static graphs via a single configuration file that is optimized before execution.

Data Transformation Languages - Uses a specialized declarative language to perform complex structural modifications on data payloads.

Real-Time Data Streaming - Processes and delivers live data streams in real-time using a specialized language for enrichment.

Stream Routing - Provides a modular system for routing data streams between diverse inputs and outputs.

Stream Transformations - Performs real-time processing of data streams using a specialized language for filtering, mapping, and enrichment.

Message Delivery Guarantees - Implements a transaction model with offsets and acknowledgments to ensure at-least-once delivery of messages.

At-Least-Once Delivery Guarantees - Ensures data is not lost during system crashes using a transaction model that guarantees at-least-once delivery.

Event Middleware - Provides the infrastructure for decoupling services by consuming and delivering messages via a transaction model.

Change Data Capture Tools - Acts as a tool to monitor and stream database modifications to external systems in real-time.

Extensible Plugin Architectures - Provides a modular architecture for adding custom functionality via external binary plugins.

Plugin-Based Architectures - Supports loading external binary modules at runtime to extend the engine with new source and sink connectors.

End-to-End Message Tracing - Generates telemetry traces to visualize the end-to-end movement of data through the processing pipeline.

Pipeline Health Monitors - Exposes health probes and emits performance metrics to external monitoring backends for operational oversight of data pipelines.

Internal Telemetry Monitors - Integrates health probes and performance metrics to expose the internal operational state of the processing pipeline.

Metric and Performance Monitors - Monitors pipeline health and performance through integrated telemetry and high-frequency numerical metrics.

Observability Data Planes - Emits performance metrics and telemetry traces to monitor the health of streaming data flows in real-time.

Data Integration - Message streaming bridge for various protocols.

benthosdevbenthos

Features

Star history