# Document deduplication > AI-ranked search results for `document duplication` on awesome-repositories.com — ordered by an LLM for relevance, best match first. 120 total matches; showing the top 12. Explore on the web: https://awesome-repositories.com/q/document-duplication **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/document-duplication).** ## Results - [arsenetar/dupeguru](https://awesome-repositories.com/repository/arsenetar-dupeguru.md) (7,631 ⭐) — dupeguru is a content-based file deduplicator and cross-platform disk cleanup tool. It functions as a local file management utility designed to identify and remove redundant files to recover disk space. The application identifies identical files across a file system by using content hashing and metadata comparison. This allows it to detect duplicates regardless of their filenames or directory locations. The software covers a range of data management capabilities, including directory scanning, content-based data deduplication, and the consolidation of redundant copies into single versions. It - [qarmin/czkawka](https://awesome-repositories.com/repository/qarmin-czkawka.md) (31,526 ⭐) — Czkawka is a cross-platform utility designed for storage optimization and filesystem maintenance. It functions as a comprehensive file analysis engine that identifies redundant data, including duplicate files, empty directories, broken symbolic links, and temporary files. By utilizing hash-based content verification, the tool ensures accurate identification of duplicates regardless of file names or metadata. The project distinguishes itself by offering both a native graphical user interface and a command-line interface, allowing for both interactive management and automated, headless system m - [google-research/deduplicate-text-datasets](https://awesome-repositories.com/repository/google-research-deduplicate-text-datasets.md) (1,273 ⭐) — Deduplicate-text-datasets is a data cleaning pipeline and text indexing engine built in Rust for large language model data preparation and corpus decontamination. It constructs massive uncompressed suffix arrays on disk using single-threaded and parallel chunk-based algorithms, enabling memory-efficient pattern searching and indexing across multi-gigabyte text corpora. The software identifies overlapping byte ranges from repeated substring matches to strip duplicate content and produce cleaned text datasets. Its search and indexing capabilities support querying corpus occurrences, locating d - [git/git](https://awesome-repositories.com/repository/git-git.md) (61,518 ⭐) — Git is a distributed version control system and command-line tool designed for tracking changes in source code and coordinating collaborative software development. It functions as a content-addressable storage platform where project data is maintained as immutable objects indexed by cryptographic hashes, ensuring data integrity and efficient deduplication. The system organizes project history as a directed acyclic graph, where each commit serves as a snapshot linked to its parent to create a verifiable timeline of modifications. The architecture distinguishes itself through an index-based sta - [kopia/kopia](https://awesome-repositories.com/repository/kopia-kopia.md) (12,612 ⭐) — Kopia is a backup utility designed to create encrypted, deduplicated, and compressed snapshots of files and directories. It functions as a client-side tool that secures data locally before transmitting it to various storage targets, ensuring that sensitive information remains protected throughout the backup process. The system utilizes content-addressable block storage and metadata-driven versioning to identify and remove redundant data across multiple snapshots. By employing a pluggable storage abstraction layer, it supports a wide range of local, network, and cloud-based storage providers, - [bup/bup](https://awesome-repositories.com/repository/bup-bup.md) (7,324 ⭐) — bup is a deduplicating backup manager and incremental backup system. It uses a Git packfile-based storage format to eliminate redundant data across files and versions, treating every incremental save as a full backup. The system provides secure remote transport interfaces for transferring and managing backup data on remote servers via SSH. It also includes a backup repository browser available as both a web interface and a filesystem mount for exploring and retrieving files from snapshots. The project covers broad capability areas including disaster recovery, repository administration, and s - [guarinogabriel/mac-cli](https://awesome-repositories.com/repository/guarinogabriel-mac-cli.md) (9,094 ⭐) — Mac-CLI is a collection of terminal commands and utilities designed for automating system management, hardware monitoring, and software configuration on macOS. It serves as a developer utility and automation tool for manipulating files, managing version control, and auditing installed packages from the command line. The project provides hardware monitoring capabilities to track real-time battery health, CPU temperature, and fan speeds. It also includes system automation tools for managing power states, disk usage, and network settings through a unified interface. The utility covers several f - [klaussinani/taskbook](https://awesome-repositories.com/repository/klaussinani-taskbook.md) (9,327 ⭐) — Taskbook is a command-line task manager and local plain-text organizer that provides a terminal user interface for managing to-do lists, project boards, and notes. It functions as a CLI Kanban board, allowing users to group actionable items and informational snippets into categorized boards. The tool emphasizes local data organization by storing information in local files with customizable storage paths. It supports a Kanban-style workflow where tasks are organized into boards and tracked through pending, in-progress, and complete states. The system includes capabilities for prioritizing tas - [ipfs/kubo](https://awesome-repositories.com/repository/ipfs-kubo.md) (17,056 ⭐) — Kubo is a peer-to-peer implementation of the InterPlanetary File System (IPFS) designed for decentralized data storage and content delivery. It uses content-addressing, directed acyclic graphs, and distributed hash tables to identify, distribute, and retrieve data across a network without relying on central servers. The project differentiates itself by providing a virtual filesystem via FUSE, which maps decentralized network namespaces to local operating system directories for direct file access. It also includes integrated HTTP gateways that translate peer-to-peer content into standard web t - [creationix/js-git](https://awesome-repositories.com/repository/creationix-js-git.md) (3,844 ⭐) — js-git is a modular JavaScript implementation of the Git protocol and object model. It provides a content-addressable storage engine that manages blobs, trees, and commits using cryptographic hashes, allowing version control functionality to operate without external system dependencies. The project features a pluggable repository interface and storage layer, enabling Git data to be persisted across diverse mediums such as browser databases or local file systems. It includes a streaming packfile processor for packing and unpacking objects to facilitate efficient data transfer. The library cov - [jarun/buku](https://awesome-repositories.com/repository/jarun-buku.md) (7,145 ⭐) — Buku is a personal bookmark manager that provides a command line interface, a portable bookmark database, and a self-hosted server for organizing web links. It functions as a command line knowledge base for saving, tagging, and searching web resources. The system features a portable, mergeable database that supports AES-256 encryption and is designed for cross-device data synchronization. It includes a RESTful API and a self-hosted web interface, allowing users to manage their collection via a browser or programmatic requests. Capabilities include automatic metadata fetching to populate page - [tagstudiodev/tagstudio](https://awesome-repositories.com/repository/tagstudiodev-tagstudio.md) (6,766 ⭐) — TagStudio is a database-driven digital asset manager designed to organize and categorize files without altering their physical location on disk. It utilizes a non-destructive metadata store to attach tags and custom fields to files via a separate database rather than modifying original file headers. The project is distinguished by a hierarchical tagging system that supports parent-child relationships and custom color palettes for visual organization. It includes a multi-format media previewer that generates thumbnails and playback for images, videos, audio, documents, and archives, as well as