zdennis/activerecord-import
Activerecord Import
ActiveRecord Import ist eine Bibliothek zum Einfügen großer Datensätze in eine Datenbank mittels ActiveRecord durch einzelne oder gebündelte Abfragen. Es fungiert als Bulk-Daten-Importer und SQL-Bulk-Insert-Tool, das darauf ausgelegt ist, den Abfrage-Overhead zu minimieren und die Schreibleistung zu erhöhen.
Das Projekt enthält eine Upsert-Engine, um Konflikte bei eindeutigen Constraints durch das Aktualisieren bestehender Datensätze oder das Ignorieren von Duplikaten zu behandeln. Es bietet zudem einen rekursiven Assoziations-Importer, der es ermöglicht, Eltern-Datensätze und deren verschachtelte Kind-Assoziationen in einem einzigen Vorgang einzufügen.
Die Bibliothek bietet eine Validierungsschicht vor dem Einfügen, um die Gültigkeit der Datensätze zu prüfen und Fehler zu melden, bevor Daten festgeschrieben werden. Sie verwaltet die groß angelegte Datenmigration durch Aufteilen der Datensätze in Batches, um den Speicherverbrauch zu kontrollieren und Limits für die Datenbank-Paketgröße zu vermeiden.
Das Tool bietet Ergebniszusammenfassungen mit Erfolgszählern und Datensatz-Identifikatoren sowie eine Fortschrittsverfolgung für Daten-Batches.
Features
- Bulk Record Insertions - Provides high-performance capabilities to insert large datasets into a database using single or batched queries.
- ActiveRecord Bulk Importers - Provides a specialized library for inserting large datasets into a database using ActiveRecord through single or batched queries.
- Batch Size Management - The project limits the number of rows processed per single statement to optimize memory and avoid database packet size limits.
- Conflict Resolution - Handles data collisions during bulk writes by updating existing records or ignoring duplicates.
- Data Upsert Operations - Manages the upsert pattern to ensure data presence and currency when importing records that may violate unique constraints.
- Upsert Engines - Implements a data import tool that handles unique constraint conflicts by updating existing records or ignoring duplicates.
- Data Validation Layers - Provides a data validation layer to enforce integrity and validity before records are persisted to the database.
- Pre-Insertion Record Validations - Checks the validity of large datasets before insertion and tracks which specific records failed the validation process.
- Pre-Insertion Validations - Runs model validation logic on each record in a collection before committing the bulk operation.
- Batch Statement Generation - Chunks large datasets into smaller groups to prevent database packet size errors and manage memory usage.
- Upsert Strategies - Implements strategies to either update existing records or ignore conflicts based on unique constraints during imports.
- Import Chunking - Splits massive datasets into smaller chunks to manage memory usage and enable progress tracking via callbacks.
- Bulk Record Validations - Verifies record validity before insertion and generates detailed reports for instances that fail the validation process.
- Multi-Format Input Support - Processes bulk data from various source types including arrays of values, hashes, and model collections.
- Import Result Summaries - Returns detailed summaries containing the count of successful inserts and identifiers for the created records.
- Batch-Driven Data Migration - Processes massive datasets in controlled batches to manage memory usage and avoid database packet size limits.
- Association Record Management - Manages the recursive insertion of parent records and their nested child associations in a single database operation.
- Association Traversals - Processes parent and child records in a sequence that maintains foreign key integrity across related tables.
- Related Record Importers - Resolves and associates related database records recursively during the bulk import process.
- Dialect-Specific SQL Generation - Generates SQL syntax tailored to specific database dialects for optimized bulk inserts and upserts.
- Import Progress Tracking - Tracks and displays the completion status of bulk imports as data batches are processed.
- ORM Extensions - Bulk inserts data into databases efficiently.
- Database Tools - Handles bulk data insertion.
