xroche/httrack
Httrack
HTTrack is a website mirroring tool that downloads entire websites to a local directory, preserving the relative link structure so the site remains fully navigable without an internet connection. It operates as an HTTP website downloader and offline browsing utility, capturing static copies of remote sites for long-term offline storage and archiving.
The tool includes a resumable download state machine that persists progress, allowing interrupted transfers to continue without re-fetching already retrieved files. It also features an incremental synchronization engine that compares local and remote file metadata to download only new or modified content during mirror updates. Bandwidth and connection throttling controls limit concurrent connections and transfer speed to reduce server load, while a robots.txt compliance layer parses and enforces exclusion rules by default.
Content filtering capabilities allow users to restrict downloads by file type, size, and modification time, and the recursive link rewriting engine rewrites absolute and relative hyperlinks in downloaded HTML to local filesystem paths for offline navigation. The shared-library core architecture exposes mirroring logic as a reusable library, enabling integration into other software or command-line automation.
Features
- Website Offline Mirroring - Downloads entire websites to local storage for offline browsing and long-term preservation.
- Static Site Archivers - Captures and preserves static copies of websites for long-term offline storage.
- Website Content Synchronization - Updates existing local mirrors by downloading only new or modified files from the source.
- Offline Browsing Tools - Creates fully navigable local copies of websites with rewritten internal links for offline access.
- Local-Link Rewriters - Rewrites absolute and relative hyperlinks in downloaded HTML to local filesystem paths for offline navigation.
- Resumable Downloads - Continues interrupted website downloads without re-downloading already retrieved files.
- Website Downloaders - Downloads web pages and assets over HTTP for offline access and archiving.
- Website Mirroring Tools - Downloads entire websites to local storage for fully navigable offline browsing with rewritten links.
- Incremental Comparison Engines - Compares local and remote file metadata to download only new or modified content during mirror updates.
- Website Mirror Refreshers - Resumes interrupted downloads and refreshes a previously mirrored site with only changed or missing files.
- Robots Exclusion Compliance - Honors robots.txt rules by default to avoid downloading disallowed pages.
- Download Scope Filters - Restricts mirrored content by file type, size, or modification time to exclude unwanted files.
- Size-Based File Filtering - Restricts downloads to only specific file types or excludes files outside a given size range during a mirror operation.
- Bandwidth Management - Controls download speed and connection rate to avoid overloading source servers.
- Connection Rate Limiting - Caps the number of simultaneous connections to a server to prevent overloading it.
- Download Scope Filters - Applies size, time, and type filters to exclude unwanted files from a mirror.
- Download Traffic Controllers - Restricts download speed to avoid saturating the network connection during mirroring.
- Shared Library Interfaces - Exposes mirroring logic as a reusable shared library for integration into other software.
- Connection Rate Limiting - Limits concurrent connections and transfer speed to reduce server load during downloads.
- System Utilities - Utility for mirroring websites for offline browsing.
- Other Language Crawlers - Tool for copying websites locally.
