bruin-data/ingestr
Ingestr
ingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations.
The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets.
The system provides capabilities for cross-database migration and cross-platform data transfer. It utilizes batch-based stream processing to move large volumes of data and employs an upsert strategy to maintain data consistency based on unique keys.
Features
- Cross-Database Data Migrations - Moves data between disparate database engines and third-party platforms without requiring custom migration code.
- Cross-Database Data Transfers - Moves data across various database engines and external platforms by streaming result sets.
- Incremental Syncing - Updates destination tables by identifying new or modified records through primary keys to maintain source state.
- ETL Workflows - Automates the extraction, transformation, and loading of data from diverse sources via a CLI.
- Data Synchronization Pipelines - Maintains an accurate reflection of source data in a destination system through automated sync pipelines.
- ETL Pipeline Tools - Functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations.
- Incremental Data Loading - Updates destination tables by processing only new or modified records to keep datasets current.
- Incremental Data Synchronization - Fetches and merges only new data records generated since the last synchronization to avoid full re-loads.
- Plugin-Based Data Source Adapters - Implements a pluggable adapter system to extract raw data from diverse databases and third-party APIs.
- Schema-Agnostic Data Mappings - Maps source fields to destination columns dynamically without requiring predefined static table definitions.
- Schema Column Mapping - Dynamically maps source fields to destination columns during data transfer between disparate systems.
- Upsert Strategies - Combines inserting new records and updating existing ones based on a unique key to maintain data consistency.
- Bulk Data Transfers via COPY - Moves large volumes of data between various databases and platforms using high-throughput transfer mechanisms.
- Database Migration CLIs - Provides a command-line interface to automate data copying and syncing between different database engines.
- Batch Data Processing - Efficiently moves large volumes of data between systems using bulk operations and small chunks.
- Incremental Sync Checkpointings - Tracks the last processed record using markers to fetch only new or updated data since the last run.
- Batch-Based Stream Processing - Processes large datasets in small chunks to maintain a low memory footprint during data transfer.
- Schema-Agnostic Ingestion - Dynamically maps source fields to destination columns without requiring predefined static table definitions.
- Upsert Operations - Implements atomic insert-or-update operations based on unique keys to maintain data consistency during transfers.
- CLI-Driven Configurations - Drives the entire data pipeline through command-line arguments instead of external configuration files.
- Data Ingestion - CLI utility for copying data between various sources.
- Data Ingestion Pipelines - CLI tool for copying data between various sources.
- Data Integration Providers - Copy data between various sources and destinations via CLI.
- Data Pipelines - CLI tool for copying data between sources and destinations.
- Developer Tools - CLI tool to copy data from any source to any destination.
