pkolaczk/fclones
Fclones
fclones is a command-line tool designed to locate identical files across a filesystem by comparing file sizes and cryptographic hashes. It functions as a parallel filesystem scanner and a deduplication utility that identifies duplicate files to reclaim disk space.
The tool distinguishes itself through a persistent hash cache system that stores hashes and metadata on disk to accelerate repeated scans. It employs a multi-phase scanning process and device-aware parallel I/O, which adjusts thread pools based on whether the storage is an SSD or HDD to maximize throughput.
Beyond discovery, the project provides methods for disk space reclamation, including redundant file removal and the substitution of duplicate files with hard links or copy-on-write reflinks. It also supports file replication analysis based on configurable replica thresholds and allows for file selection via attribute-based filtering and exclusion patterns.
Findings can be exported to text, CSV, or JSON formats, and the tool includes a dry-run mode to preview filesystem changes before they are executed.
Features
- Duplicate File Finders - Scans storage to identify files with identical content for deduplication and review.
- Content-Based Deduplication - Eliminates redundant data by identifying identical files through cryptographic content hashing.
- Incremental Scan Caches - Stores file hashes and metadata on disk to accelerate repeated scans by skipping unchanged files.
- Storage Reclamation Utilities - Reclaims disk space by identifying and removing unnecessary duplicate files.
- Hard Link Management - Replaces duplicate file copies with hard links to the same inode to reclaim disk space.
- Parallel Filesystem Scanning - Scans large volumes of data using parallel I/O and hardware-aware scheduling to find duplicates efficiently.
- Link-Based File Substitution - Substitutes duplicate files with hard or symbolic links to save disk space while maintaining accessibility.
- Reflink-Based Deduplication - Reclaims disk space by using filesystem reflinks to share data blocks between duplicate files.
- Filesystem Scanners - Implements a high-performance scanner that traverses storage media to identify duplicate files.
- Reflink Deduplication - Uses filesystem copy-on-write reflinks to share data blocks between identical files without deleting them.
- Multi-Phase Hashing Strategies - Minimizes disk I/O by filtering files via size and partial hashes before computing full cryptographic signatures.
- Hardware-Aware Scheduling - Adjusts thread pool concurrency and I/O scheduling based on whether the storage is an SSD or HDD to maximize throughput.
- Persistent Hash Caching - Stores file hashes on disk and verifies modification times and sizes to avoid re-hashing unchanged files.
- Redundant File Scanning - Scans the filesystem to identify and remove duplicate files to reclaim storage space.
- Replica Count Thresholds - Allows users to set minimum or maximum replica counts per content group for deduplication and replication analysis.
- Multi-Phase Hashing - Scans files in stages by size, partial hashes, and then full hashes to minimize I/O and CPU usage.
- Dry Run Simulations - Provides a preview mode to simulate and verify filesystem changes before they are executed.
- Device-Aware I/O Scheduling - Adjusts thread-pool sizes and I/O concurrency based on storage device type to maximize throughput.
- Scan Concurrency Control - Manages the number of threads used for scanning and I/O to balance performance and system load.
- File Hash Verification - Stores computed file hashes and validates them against metadata to avoid re-hashing unchanged files.
- File Redundancy Analysis - Identifies files that are under-replicated or over-replicated based on a configurable threshold to ensure data redundancy.
- System Utilities - Efficient duplicate file finder.
