elasticsearch-dump/elasticsearch-dump
Elasticsearch Dump
elasticsearch-dump is a command-line tool for importing and exporting data, mappings, and settings between Elasticsearch and OpenSearch clusters. It serves as a backup and restore system, a data transfer utility for cross-cluster migrations, and a bridge for streaming search index data to and from S3-compatible storage.
The tool functions as a JSON index data transformer, allowing the use of custom JavaScript modules to modify or anonymize documents in flight during the migration process. It also enables the creation of search bodies using external JavaScript templates to customize the data being retrieved during export.
Capabilities cover a wide range of data movement tasks, including moving indices between servers, synchronizing with AWS S3 and MinIO buckets, and importing data from CSV files. The tool can back up index components such as mappings, analyzers, templates, and aliases to local JSON files.
The tool is available as a Docker image for containerized execution.
Features
- Cross-Database Data Migrations - Moves data directly from one search cluster to another to facilitate environment migrations.
- Elasticsearch Data Migration - Moves search indices and mappings between different Elasticsearch or OpenSearch clusters.
- Indexing Transformation Pipelines - Allows modifying or anonymizing documents using custom JavaScript modules during the migration process.
- Backup and Recovery - Provides tools for creating snapshots of OpenSearch index data and restoring them to prevent data loss.
- OpenSearch Data Transfer Utilities - Moves documents and index configurations between OpenSearch servers using files or network connections.
- Index Migration - Loads data from an external source into a search index to restore backups or migrate information.
- Index Component Imports - Loads index data, mappings, templates, and aliases from local JSON files into a server.
- Index Data Export - Pulls data from a search index and saves it to an external destination for backup or migration.
- In-Flight Data Modification - Executes custom JavaScript modules to change or anonymize documents while they are being transferred.
- Transformation Hooks - Provides custom JavaScript hooks to modify or anonymize documents in flight during the migration process.
- Search Index Backups - Saves search index documents, mappings, analyzers, templates, and aliases to local files to prevent data loss.
- Search Index Transfer - Transfers data, mappings, and analyzers between search indices using URLs, files, or standard input and output.
- Search Index Migrations - Copies index data, mappings, analyzers, and settings from one server to another to relocate information.
- Document Transformation Modules - Provides custom JavaScript modules to modify or anonymize documents on the fly as they move between source and destination.
- Backup and Restore - Saves index components and documents to local JSON files or S3-compatible storage for recovery.
- Parallel Worker Execution - Utilizes a worker pool to execute multiple import or export processes concurrently to increase throughput.
- CSV Import Managers - Parses CSV files and loads content into a search index using custom delimiters and row skipping.
- Bulk Data Migrations - Runs multiple dump or load processes across several indices simultaneously to increase transfer speeds.
- Query Result Exporters - Saves the results of specific search queries, defined by JSON bodies or external files, to an external destination.
- Index Pattern Matching - Selects specific indices for migration using regular expressions or custom matcher modules to target specific data.
- Pluggable Storage Backends - Implements a common interface to abstract read and write operations across local files, S3 buckets, and standard I/O.
- Query Generation Templates - Uses external JavaScript templates to dynamically construct search bodies for customized data extraction.
- Search Data Extraction - Pulls specific documents or query results from a search index into external files for analysis or auditing.
- Search Query Construction - Allows creating search bodies during the export process using external JavaScript templates to customize retrieved data.
- Stream-Based Data Pipelines - Processes records sequentially through read and write streams to handle large datasets without loading everything into memory.
- S3-Compatible Data Synchronization - Enables streaming search index data directly to and from AWS S3 and MinIO buckets.
- S3 Search Data Bridges - Connects search index data streams directly to or from AWS S3 and MinIO buckets.
- Local Worker Pools - Distributes the processing of multiple indices across a pool of concurrent workers to increase transfer speed.
- Bulk Operation Batching - Groups multiple documents into single HTTP requests to maximize throughput during index import and export.
