elasticsearch-dump/elasticsearch-dump

Features

• Cross-Database Data Migrations - Moves data directly from one search cluster to another to facilitate environment migrations.
• Elasticsearch Data Migration - Moves search indices and mappings between different Elasticsearch or OpenSearch clusters.
• Indexing Transformation Pipelines - Allows modifying or anonymizing documents using custom JavaScript modules during the migration process.
• Backup and Recovery - Provides tools for creating snapshots of OpenSearch index data and restoring them to prevent data loss.
• OpenSearch Data Transfer Utilities - Moves documents and index configurations between OpenSearch servers using files or network connections.
• Index Migration - Loads data from an external source into a search index to restore backups or migrate information.
• Index Component Imports - Loads index data, mappings, templates, and aliases from local JSON files into a server.
• Index Data Export - Pulls data from a search index and saves it to an external destination for backup or migration.
• In-Flight Data Modification - Executes custom JavaScript modules to change or anonymize documents while they are being transferred.
• Transformation Hooks - Provides custom JavaScript hooks to modify or anonymize documents in flight during the migration process.
• Search Index Backups - Saves search index documents, mappings, analyzers, templates, and aliases to local files to prevent data loss.
• Search Index Transfer - Transfers data, mappings, and analyzers between search indices using URLs, files, or standard input and output.
• Search Index Migrations - Copies index data, mappings, analyzers, and settings from one server to another to relocate information.
• Document Transformation Modules - Provides custom JavaScript modules to modify or anonymize documents on the fly as they move between source and destination.
• Backup and Restore - Saves index components and documents to local JSON files or S3-compatible storage for recovery.
• Parallel Worker Execution - Utilizes a worker pool to execute multiple import or export processes concurrently to increase throughput.
• CSV Import Managers - Parses CSV files and loads content into a search index using custom delimiters and row skipping.
• Bulk Data Migrations - Runs multiple dump or load processes across several indices simultaneously to increase transfer speeds.
• Query Result Exporters - Saves the results of specific search queries, defined by JSON bodies or external files, to an external destination.
• Index Pattern Matching - Selects specific indices for migration using regular expressions or custom matcher modules to target specific data.
• Pluggable Storage Backends - Implements a common interface to abstract read and write operations across local files, S3 buckets, and standard I/O.
• Query Generation Templates - Uses external JavaScript templates to dynamically construct search bodies for customized data extraction.
• Search Data Extraction - Pulls specific documents or query results from a search index into external files for analysis or auditing.
• Search Query Construction - Allows creating search bodies during the export process using external JavaScript templates to customize retrieved data.
• Stream-Based Data Pipelines - Processes records sequentially through read and write streams to handle large datasets without loading everything into memory.
• S3-Compatible Data Synchronization - Enables streaming search index data directly to and from AWS S3 and MinIO buckets.
• S3 Search Data Bridges - Connects search index data streams directly to or from AWS S3 and MinIO buckets.
• Local Worker Pools - Distributes the processing of multiple indices across a pool of concurrent workers to increase transfer speed.
• Bulk Operation Batching - Groups multiple documents into single HTTP requests to maximize throughput during index import and export.

Star history