benthosdev/benthos

Features

• Data Ingestion and Integration - Moves and integrates data across different platforms by consuming from sources, transforming, and delivering to destinations.
• Data Integration Pipelines - The product enables moving data between various sources and sinks using a declarative pipeline configuration defined within a single file.
• Stream Processing Engines - Provides a declarative engine for routing, transforming, and connecting high-velocity data streams between diverse sources and sinks.
• Data Pipelines and ETL - Functions as a framework for building data transformation and integration pipelines with built-in filtering.
• Change Data Capture - Streams real-time database modifications as discrete events for downstream processing.
• Declarative Workflow Definitions - Simplifies stream management by defining data movement and transformation via a single configuration file.
• Declarative Pipeline Construction - Defines data workflows as static graphs via a single configuration file that is optimized before execution.
• Data Transformation Languages - Uses a specialized declarative language to perform complex structural modifications on data payloads.
• Real-Time Data Streaming - Processes and delivers live data streams in real-time using a specialized language for enrichment.
• Stream Routing - Provides a modular system for routing data streams between diverse inputs and outputs.
• Stream Transformations - Performs real-time processing of data streams using a specialized language for filtering, mapping, and enrichment.
• Message Delivery Guarantees - Implements a transaction model with offsets and acknowledgments to ensure at-least-once delivery of messages.
• At-Least-Once Delivery Guarantees - Ensures data is not lost during system crashes using a transaction model that guarantees at-least-once delivery.
• Event Middleware - Provides the infrastructure for decoupling services by consuming and delivering messages via a transaction model.
• Change Data Capture Tools - Acts as a tool to monitor and stream database modifications to external systems in real-time.
• Extensible Plugin Architectures - Provides a modular architecture for adding custom functionality via external binary plugins.
• Plugin-Based Architectures - Supports loading external binary modules at runtime to extend the engine with new source and sink connectors.
• End-to-End Message Tracing - Generates telemetry traces to visualize the end-to-end movement of data through the processing pipeline.
• Pipeline Health Monitors - Exposes health probes and emits performance metrics to external monitoring backends for operational oversight of data pipelines.
• Internal Telemetry Monitors - Integrates health probes and performance metrics to expose the internal operational state of the processing pipeline.
• Metric and Performance Monitors - Monitors pipeline health and performance through integrated telemetry and high-frequency numerical metrics.
• Observability Data Planes - Emits performance metrics and telemetry traces to monitor the health of streaming data flows in real-time.
• Data Integration - Message streaming bridge for various protocols.

Star history