Benthos

Benthos is a declarative stream processor and data integration pipeline used to route, transform, and filter information between disparate services. It functions as an at-least-once message broker and change data capture engine, using a transaction model to guarantee message delivery despite system crashes or server faults.

The system is defined by an observability-first approach, featuring built-in HTTP health probes, performance metrics export, and distributed request flow tracing. It utilizes a plugin architecture that allows the core engine to be extended with custom binaries for new input and output connectors.

The project manages data flow through a single configuration file to define pipelines. These pipelines support real-time database change events, buffered stream processing, and data transformation using a specialized mapping language.

Features

Data Streaming - Provides a declarative system for routing and moving real-time data streams between disparate services.

Change Data Capture - Streams real-time database change events from supported databases into downstream processing pipelines.

Data Pipeline Configurations - Uses a single declarative configuration file to define the connection of diverse data sources and sinks.

Data Stream Processors - Offers a declarative stream processing architecture to route, transform, and filter real-time data flows.

Message Brokers - Acts as a data transport system that guarantees message delivery via a transaction model.

Stream Transformations - Modifies, enriches, and filters information in transit as it moves from a source to a destination.

At-Least-Once Delivery Guarantees - Ensures messages are delivered at least once using a transaction model that survives system crashes and faults.

Data Mapping Languages - Provides a specialized mapping language to filter and enrich data payloads as they move through the pipeline.

Data Integration - Facilitates the connection and exchange of information between disparate software systems and data sources.

Data Pipelines - Provides composable stages for moving and transforming data between disparate services.

Stream Buffering Pipelines - Utilizes internal buffers to decouple source ingestion from destination output across pipeline stages.

Plugin Architectures - Supports integrating custom plugins to add specialized capabilities and new behaviors to the core engine.

Plugin-Based Architectures - Allows the core engine to be extended with custom binaries for new input and output connectors.

End-to-End Message Tracing - Provides telemetry traces to visualize the end-to-end path of data as it moves through processing pipelines.

Pipeline Health Monitors - Tracks the operational health and performance of data streams through telemetry traces and monitoring backends.

Internal Telemetry Monitors - Exposes internal health and performance metrics via HTTP endpoints and tracing for external monitoring.

Observability Tools - Includes built-in HTTP health checks, performance metrics, and distributed tracing for operational monitoring.

Data Processing and Streams - Stream processing engine.

Streaming Libraries - High-performance message streaming service for transformations.

Messaging Systems - Message stream processing across protocols.

Jeffailbenthos

Features

Star history