peak/s5cmd

Features

• Object Storage - Moves large volumes of files between local filesystems and remote object storage using parallel workers.
• S3 File Transfer Tools - Functions as a high-performance utility for uploading, downloading, and copying files between local systems and S3-compatible storage.
• Cloud Storage Synchronization - Syncs local directories to remote cloud providers while preserving structure and maintaining consistency.
• Bucket Management - Provides administrative controls for creating, removing, and listing storage buckets across compatible services.
• S3 Storage Management - Implements comprehensive management of files and buckets across S3-compatible object storage services.
• SSH Directory Syncs - Mirrors local directories to remote storage by copying updated files and removing extraneous items.
• Parallel Command Execution - Enables concurrent execution of thousands of object management tasks from command files or piped streams.
• Directory Synchronization - Mirrors local directories to remote storage by copying updated files and removing extraneous items to maintain consistency.
• High-Performance Data Transfer - Provides high-performance data movement between local systems and cloud storage using parallel workers to saturate available bandwidth.
• S3 Batch Operations - Executes massive-scale batch operations across millions of S3 objects in a single request.
• Concurrent Request Pools - Uses concurrent request pools to distribute object management tasks across multiple goroutines for maximum throughput.
• Object Operation Filtering - Allows refining the target set of files for any operation using inclusion or exclusion flags.
• S3 Querying - Executes SQL expressions against stored JSON records to retrieve filtered results without downloading entire files.
• Bulk Object Deletions - Removes multiple objects using batch requests to increase execution speed and reduce network overhead.
• SQL Query Interfaces - Provides a SQL interface to filter and retrieve specific JSON records directly from distributed object storage.
• Direct Object Store Querying - Executes SQL expressions against stored JSON records to retrieve filtered results without downloading entire files.
• Request Batching - Groups multiple object operations into single API calls to reduce network round-trips and increase throughput.
• Stream-Based Data Pipelines - Implements processing architectures that pipe data from standard input directly to storage without intermediate local files.
• File Pattern Matching - Provides utilities for matching file paths using wildcard and recursive glob patterns to refine operation targets.
• Parallel Directory Scanning - Uses multi-threading to accelerate the process of scanning local directories and remote object lists.
• Direct-to-Remote Streaming - Streams data from standard input directly to a remote object to enable on-the-fly transformation.
• Data Integrity Checksums - Calculates and compares content hashes during transmission to ensure data integrity before finalizing remote objects.
• File Integrity Verifiers - Checks content checksums during transmission to ensure files are not corrupted before finalizing in storage.
• Infrastructure Management - High-performance S3 and filesystem execution tool.
• Infrastructure Tools - Fast S3 and local filesystem command-line tool.

Star history