linkedin/databus
Databus
Databus is a distributed change data capture system and database transaction log streamer. It extracts data modifications from source logs and streams them to downstream consumers to keep secondary data stores and caches consistent.
The system functions as a source-agnostic data pipeline and distributed event relay. It uses a network of processes to decouple database sources from consumers, routing change streams across a network to multiple targets regardless of the original source system.
The project covers real-time data synchronization and distributed pipeline monitoring. This includes the ability to capture database changes, distribute those streams via relays, and track connection requests to debug network connectivity.
Features
- Change Data Capture - Extracts data modifications from transaction logs to maintain a consistent source of truth across secondary data stores.
- Source-Agnostic Data Pipelines - Provides a distributed architecture that routes captured database events to multiple targets regardless of the original source system.
- Distributed Data Pipelines - Implements a distributed data pipeline that moves captured database changes through a network of relays to maintain consistency.
- Change Data Capture Systems - Streams database modifications from source logs to downstream consumers through a decoupled delivery pipeline.
- Distributed Event Distribution - Routes captured data changes through a relay system to decouple primary sources from multiple independent consumers.
- Change - Decouples database sources from consumers by routing change streams through intermediate relay processes.
- Change Data Relays - Routes captured database changes through relay processes to decouple sources from multiple downstream consumers.
- Real-time Data Synchronization - Moves transaction log updates to search indexes or caches to maintain consistency across a distributed architecture.
- Database Change Streaming - Moves captured database changes from a source to downstream applications through a distributed delivery pipeline.
- Event Relay Networks - Implements a network of relay processes that decouple database sources from consumers by routing change streams.
- Transaction Log Parsers - Extracts database modifications by parsing transaction log trail files and converting them into a portable stream format.
- Change Data Capture Streams - Routes captured data changes from primary sources to search indexes or caches to keep them consistent.
- Database Transaction Log Parsers - Extracts data changes from transaction logs to keep secondary data stores and caches consistent.
- Agnostic Event Formats - Standardizes database changes into a generic event format to enable a source-agnostic data pipeline.
- Data Pipelines - Source-agnostic distributed change data capture system.
