zdennis/activerecord-import
Activerecord Import
ActiveRecord Import este o bibliotecă pentru inserarea seturilor mari de date într-o bază de date folosind ActiveRecord prin interogări unice sau grupate. Funcționează ca un importator de date în masă și instrument de inserare SQL în masă conceput pentru a minimiza overhead-ul interogărilor și a crește performanța de scriere.
Proiectul include un motor de upsert pentru a gestiona conflictele de constrângeri unice prin actualizarea înregistrărilor existente sau ignorarea duplicatelor. De asemenea, dispune de un importator de asocieri recursiv care permite inserarea înregistrărilor părinte și a asocierilor copil imbricate într-o singură operațiune.
Biblioteca oferă un strat de validare pre-inserare pentru a verifica validitatea înregistrărilor și a raporta eșecurile înainte de a comite datele. Gestionează migrarea datelor la scară largă prin împărțirea seturilor de date în loturi pentru a controla utilizarea memoriei și a evita limitele de dimensiune a pachetelor bazei de date.
Instrumentul oferă rezumate ale rezultatelor care conțin numărul de succese și identificatorii înregistrărilor, alături de urmărirea progresului pentru loturile de date.
Features
- Bulk Record Insertions - Provides high-performance capabilities to insert large datasets into a database using single or batched queries.
- ActiveRecord Bulk Importers - Provides a specialized library for inserting large datasets into a database using ActiveRecord through single or batched queries.
- Batch Size Management - The project limits the number of rows processed per single statement to optimize memory and avoid database packet size limits.
- Conflict Resolution - Handles data collisions during bulk writes by updating existing records or ignoring duplicates.
- Data Upsert Operations - Manages the upsert pattern to ensure data presence and currency when importing records that may violate unique constraints.
- Upsert Engines - Implements a data import tool that handles unique constraint conflicts by updating existing records or ignoring duplicates.
- Data Validation Layers - Provides a data validation layer to enforce integrity and validity before records are persisted to the database.
- Pre-Insertion Record Validations - Checks the validity of large datasets before insertion and tracks which specific records failed the validation process.
- Pre-Insertion Validations - Runs model validation logic on each record in a collection before committing the bulk operation.
- Batch Statement Generation - Chunks large datasets into smaller groups to prevent database packet size errors and manage memory usage.
- Upsert Strategies - Implements strategies to either update existing records or ignore conflicts based on unique constraints during imports.
- Import Chunking - Splits massive datasets into smaller chunks to manage memory usage and enable progress tracking via callbacks.
- Bulk Record Validations - Verifies record validity before insertion and generates detailed reports for instances that fail the validation process.
- Multi-Format Input Support - Processes bulk data from various source types including arrays of values, hashes, and model collections.
- Import Result Summaries - Returns detailed summaries containing the count of successful inserts and identifiers for the created records.
- Batch-Driven Data Migration - Processes massive datasets in controlled batches to manage memory usage and avoid database packet size limits.
- Association Record Management - Manages the recursive insertion of parent records and their nested child associations in a single database operation.
- Association Traversals - Processes parent and child records in a sequence that maintains foreign key integrity across related tables.
- Related Record Importers - Resolves and associates related database records recursively during the bulk import process.
- Dialect-Specific SQL Generation - Generates SQL syntax tailored to specific database dialects for optimized bulk inserts and upserts.
- Import Progress Tracking - Tracks and displays the completion status of bulk imports as data batches are processed.
- ORM Extensions - Bulk inserts data into databases efficiently.
- Database Tools - Handles bulk data insertion.
