peak/s5cmd
S5cmd
s5cmd is a command line interface for high-performance data transfers and management tasks across S3 compatible storage services. It functions as a parallel data transfer tool and bucket synchronization utility, designed to accelerate the uploading and downloading of large volumes of files using concurrent workers.
The tool acts as a batch command processor capable of executing thousands of object management operations in parallel from command files or piped streams. It also serves as an S3 Select query client, allowing the execution of SQL expressions against stored JSON records to retrieve filtered results without downloading entire files.
Broad capabilities include the synchronization of local directories to remote storage, bulk object deletion, and the management of storage buckets. The utility also supports data streaming from standard input, storage usage calculations, and the application of object metadata and access control lists.
Features
- Object Storage - Moves large volumes of files between local filesystems and remote object storage using parallel workers.
- S3 File Transfer Tools - Functions as a high-performance utility for uploading, downloading, and copying files between local systems and S3-compatible storage.
- Cloud Storage Synchronization - Syncs local directories to remote cloud providers while preserving structure and maintaining consistency.
- Bucket Management - Provides administrative controls for creating, removing, and listing storage buckets across compatible services.
- S3 Storage Management - Implements comprehensive management of files and buckets across S3-compatible object storage services.
- SSH Directory Syncs - Mirrors local directories to remote storage by copying updated files and removing extraneous items.
- Parallel Command Execution - Enables concurrent execution of thousands of object management tasks from command files or piped streams.
- Directory Synchronization - Mirrors local directories to remote storage by copying updated files and removing extraneous items to maintain consistency.
- High-Performance Data Transfer - Provides high-performance data movement between local systems and cloud storage using parallel workers to saturate available bandwidth.
- S3 Batch Operations - Executes massive-scale batch operations across millions of S3 objects in a single request.
- Concurrent Request Pools - Uses concurrent request pools to distribute object management tasks across multiple goroutines for maximum throughput.
- Object Operation Filtering - Allows refining the target set of files for any operation using inclusion or exclusion flags.
- S3 Querying - Executes SQL expressions against stored JSON records to retrieve filtered results without downloading entire files.
- Bulk Object Deletions - Removes multiple objects using batch requests to increase execution speed and reduce network overhead.
- SQL Query Interfaces - Provides a SQL interface to filter and retrieve specific JSON records directly from distributed object storage.
- Direct Object Store Querying - Executes SQL expressions against stored JSON records to retrieve filtered results without downloading entire files.
- Request Batching - Groups multiple object operations into single API calls to reduce network round-trips and increase throughput.
- Stream-Based Data Pipelines - Implements processing architectures that pipe data from standard input directly to storage without intermediate local files.
- File Pattern Matching - Provides utilities for matching file paths using wildcard and recursive glob patterns to refine operation targets.
- Parallel Directory Scanning - Uses multi-threading to accelerate the process of scanning local directories and remote object lists.
- Direct-to-Remote Streaming - Streams data from standard input directly to a remote object to enable on-the-fly transformation.
- Data Integrity Checksums - Calculates and compares content hashes during transmission to ensure data integrity before finalizing remote objects.
- File Integrity Verifiers - Checks content checksums during transmission to ensure files are not corrupted before finalizing in storage.
- إدارة البنية التحتية - High-performance S3 and filesystem execution tool.
- Infrastructure Tools - Fast S3 and local filesystem command-line tool.
