redpanda-data/connect
Connect
Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments.
The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pipeline that exports metrics and execution traces using the OpenTelemetry standard.
The system covers a broad range of integration capabilities, including cloud data warehousing for services like BigQuery and Iceberg, as well as SQL data management and cloud storage integration. It supports advanced data operations such as Grok text processing, schema registry integration, and broker message routing for distributing data to multiple outputs.
Configuration is managed through structured files, with available utilities for configuration schema validation and natural language pipeline generation.
Features
- Real-Time Data Integration Platforms - Provides a platform for synchronizing live data across heterogeneous storage and analytical environments using declarative pipelines.
- Change Data Capture - Provides real-time streaming of database modifications by polling transaction logs or using change-tracking mechanisms.
- Change Data Capture Tools - Monitors and streams database modifications to Kafka topics and other external systems in real-time.
- Kafka Connectors - Provides a framework for building declarative data pipelines that move and transform messages between Kafka and external sources.
- Stream Management - Manages the flow of messages through Kafka topics with built-in support for batching and consumer group control.
- Data Integration Pipelines - Provides systems that orchestrate the movement and routing of data streams between diverse sources and sinks.
- Data Pipeline Configurations - Uses structured files to declaratively define the topology and data flow between sources and sinks.
- Stream Processing Engines - Operates as a system that performs continuous computation, transformation, and enrichment on real-time data streams.
- Data Transformation Languages - Provides Bloblang, a dedicated declarative language for mutating, filtering, and reshaping message payloads.
- Kafka Client Integrations - Integrates with Apache Kafka clusters as a producer and consumer with batching and consumer group control.
- Kafka Connect Frameworks - Functions as a distributed system for connecting Kafka clusters to databases and cloud storage via source and sink connectors.
- Message Routing - Provides advanced routing capabilities to distribute data streams to multiple targets using fan-out, sequential, or fallback patterns.
- Stream Transformations - Implements real-time processing of data streams to apply mutations and reshaping using a dedicated mapping language.
- Data Mapping Languages - Uses a dedicated domain-specific language to filter, mutate, and reshape streaming data payloads.
- Data Enrichment - Enriches real-time data streams by sending payloads to external HTTP endpoints and integrating the responses.
- Iceberg Catalog Exporters - Provides utilities for transferring processed data into Apache Iceberg-based catalogs with automatic schema evolution.
- Data Warehousing - Syncs streaming data to large-scale analytics warehouses and table catalogs for high-performance analytical queries.
- BigQuery Connectors - Ships specialized modules for synchronizing data to BigQuery analytics warehouses with support for upsert modes.
- Cloud Object Storage - Supports reading and writing unstructured data as blobs across various cloud storage providers.
- Schema Registries - Encodes and decodes messages by fetching versioned schema definitions from a remote schema registry.
- SQL Data Retrieval - Extracts, filters, and aggregates data from relational tables using standard SQL query language.
- Transactional Delivery Guarantees - Ensures message reliability and prevents data loss using a local transaction model for managing offsets and state.
- At-Least-Once Delivery Guarantees - Ensures every message is delivered at least once using an in-process transaction model to prevent data loss.
- Embedded Wasm Runtimes - Runs custom processing logic within the data pipeline using an embedded WebAssembly runtime.
- Plugin Architectures - Implements a plugin architecture allowing new source and sink capabilities to be added as compiled modules.
- Plugin Architectures - Provides an extensibility framework for implementing custom source and sink capabilities using compiled plugins.
- Schema Registries - Integrates with remote schema registries to decode and encode message payloads using versioned definitions.
- Wasm-Based Plugins - Executes custom processing logic via WebAssembly modules for safe, portable, and sandboxed pipeline extensions.
- End-to-End Message Tracing - Implements end-to-end tracing to visualize the complete flow of messages as they move through the data pipeline.
- Pipeline Health Monitors - Includes specialized monitors to track the execution status, timing, and operational health of data pipelines.
- OpenTelemetry Exporters - Exports pipeline telemetry and execution traces using standard OpenTelemetry protocols for end-to-end visibility.
- Performance Metrics Exporters - Exposes pipeline performance telemetry in standard formats for integration with external monitoring backends.
