Open-source alternatives to Httrack

Okdownload is a multi-threaded HTTP download engine and transfer library designed for concurrent file downloading. It functions as a resumable file downloader that splits single files into multiple blocks for simultaneous processing to increase network throughput and transfer speeds. The system distinguishes itself through a builder-based component injection framework and a strategy pattern that allows for custom download engines. Developers can replace default storage and connection modules or override the logic used for block splitting and filename generation. The library provides comprehe

ai.robots.txt is an AI crawler governance tool and robots exclusion protocol manager designed to control how artificial intelligence models discover and consume website data. It functions as a framework for managing bot access lists and blocking automated AI agents from scraping web pages. The project provides an AI content licensing framework that allows site owners to define terms and payment processing for AI companies wishing to access site content. This enables a content monetization strategy by establishing structured rules for the right to use data for model training. The system cover

This is an archived snapshot of the first version of the Coding Train website, preserving its original static structure and content. It serves as a static site archive, storing pre-rendered HTML pages and assets for direct delivery without server-side processing. The site organizes tutorial and challenge content as individual HTML files within a directory structure that mirrors the site hierarchy, using hyperlinks and browser-native navigation to move between pages. It manages the entire website source code and content revisions using Git for collaborative contribution and history tracking, a

This project is an Objective-C HTTP networking library that provides an object-oriented wrapper for CFNetwork to execute synchronous and asynchronous requests on macOS and iOS. It functions as a framework for managing HTTP sessions, performing resumable file transfers, and handling offline content caching. The library distinguishes itself through a resumable file transfer manager that tracks byte offsets and utilizes disk-backed streaming for large uploads and downloads. It also includes a secure credential manager that integrates with the system keychain to handle client certificates and aut

FastHub is a native Android mobile client designed for managing GitHub repositories, issues, and pull requests. It provides a visual interface for interacting with Git version control, acting as a repository manager for editing files, tracking project milestones, and handling account administration from a handheld device. The application supports multi-account authentication, allowing users to connect via tokens or passwords to access different user profiles and enterprise accounts. It distinguishes itself with offline repository browsing, utilizing local database caching to view project data

This project is a static site forum archive and housing market dataset. It serves as a read-only repository of preserved community discussions focused on residential property pricing and real estate trends, stored as a collection of Markdown files to ensure long-term data preservation and portability. The system functions as a web content preservation tool that converts historical forum threads into pre-rendered HTML files for serverless access. This approach organizes forum content archiving and housing market analysis into a permanent digital format to prevent data loss. The archive is bui

XDM is a multi-threaded download manager and HTTP download accelerator designed to increase file transfer speeds by splitting files into multiple segments for simultaneous retrieval. It functions as a browser-integrated download client that uses extensions to intercept and capture web download requests. The software includes a streaming video downloader capable of intercepting streaming protocols to save online video content as local files. Additionally, it features a video format converter to change file containers and codecs for compatibility with various devices. The application provides

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

proxyee-down is an HTTP proxy download manager and multi-threaded download client. It is designed to route file download requests through an HTTP proxy server to accelerate data retrieval and bypass network restrictions. The tool increases transfer speeds by splitting large files into smaller chunks and downloading them concurrently across multiple TCP connections. It features the ability to update expired or changed source URLs for existing tasks, allowing interrupted downloads to recover and resume without losing previously downloaded segments. The system provides controls for manual downl

gdown is a command-line tool that downloads public files and folders from Google Drive without requiring authentication. It bypasses the mandatory virus-scan warning page to retrieve large files that conventional download tools block, and can resume interrupted transfers using HTTP range requests. Beyond simple file downloads, gdown can recursively download entire folder hierarchies while preserving the local directory structure. It lists the contents of a public folder as structured JSON without downloading the files themselves, and resolves a file's real name and extension without retrievin

Pan-light is a tool for managing and downloading files from Baidu network storage. It provides high-speed file downloading and remote media streaming capabilities. The application uses concurrent segmented transfers to maximize network throughput and employs disk-persisted state tracking to resume interrupted downloads. It also includes a local proxy to bypass hotlinking protections, allowing remote videos to be streamed directly into a media player. The project features a cross-platform graphical user interface built with Go and Qt. Additionally, it supports remote software previews by runn

OkHttp-OkGo is a network request framework that wraps OkHttp to simplify the process of sending and receiving data from web services. It provides a structured system for managing asynchronous data streams and concurrency through integration with RxJava. The project includes dedicated managers for batch file transfers, enabling the simultaneous upload and download of multiple files. The download system supports breakpoint-based management, allowing interrupted transfers to be paused and resumed from the last saved position. Additional capabilities include a custom HTTP cache manager for defin

Clumsy is a Windows network fault injection tool that intercepts system-wide network traffic at the kernel level using the WinDivert driver, without requiring proxy configuration or application code changes. It captures and manipulates packets for all running applications, including localhost connections, enabling controlled testing of application behavior under degraded network conditions. The tool provides comprehensive network impairment capabilities including latency simulation, bandwidth throttling, packet loss, corruption, duplication, and reordering. Users can apply these impairments s

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Nuke is an image loading system designed to fetch, cache, and render images and short video clips within a user interface. It functions as an asset caching engine and a resumable download manager to handle the retrieval of remote media files. The framework includes a request priority manager to coordinate duplicate requests and order loading based on visual importance. It employs an image prefetching strategy to load visuals into memory before they are required and uses an LRU asset cache to reduce network traffic. The system covers broader capabilities in traffic management and data storage

rTorrent is a headless BitTorrent client daemon and manager designed for Unix-like systems. It operates as a background process to handle the downloading and uploading of files via the BitTorrent protocol. The application provides a text-based user interface using ncurses for real-time updates and keyboard navigation in a console. It also exposes a socket-based remote control interface, allowing external clients to send commands and query status via a network protocol. The software covers core torrent management capabilities, including file priority configuration, torrent import, and list fi

Croc is a command-line utility for sending files and folders between computers using end-to-end encrypted peer-to-peer connections. It employs elliptic curve encryption and key agreement to secure data transmission between remote endpoints. The tool allows users to coordinate transfers using a shared code phrase and supports the operation of custom relay servers to facilitate connections without relying on public infrastructure. It also includes a proxy client to route encrypted traffic through SOCKS5 proxies. Additional capabilities include resumable data transmission for unstable connectio

This project is a comprehensive knowledge base and study resource designed for mastering technical interviews. It provides structured guides, roadmaps, and curricula focused on data structures, algorithms, system design, and frontend engineering to help candidates prepare for software engineering screenings. The repository distinguishes itself by offering a holistic approach to professional advancement. Beyond technical drills, it includes a career development handbook covering resume optimization, salary benchmarking, and strategic negotiation coaching. It also provides detailed methodologie

This project is a tool for transforming live HTML web pages into editable Figma design layers by processing the document object model and CSS. It functions as a web-to-design converter and HTML-to-layer transformer that translates browser elements into design tool primitives. The tool captures live webpage sections across multiple screen widths to preserve responsive layout behaviors during the import process. It also includes a visual content editor that identifies live website components, allowing for the direct modification and publication of content entries from a browser view. The syste

dn is a self-hosted personal web archiving system that automatically intercepts and stores web pages on a local device. It uses a proxy-based request interception model to capture browser traffic and save content for offline access without an internet connection. The system features a local full-text search engine that indexes all saved page content for information retrieval across the collection. It includes a dedicated browser interface that simulates online connectivity to serve archived files, mimicking the original live web environment. Administrative control is provided through a web-b

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

Droid-ify is an Android app installer and client for browsing, installing, and updating open-source software from F-Droid repositories. It functions as a searchable repository browser that enables the discovery and deployment of community-maintained software packages. The project supports diverse installation pathways, routing requests through different system conduits based on the device's privilege level. It provides support for root access, session-based modes, and Shizuku to bypass standard system permission limitations. The system includes a background service that monitors repositories

PythonSpiderNotes is a comprehensive instructional resource and framework for building web crawlers and extracting data using the Python programming language. It provides a set of methods for parsing unstructured HTML and JSON data into structured formats for persistent storage. The project includes detailed guides and tutorials on browser automation for retrieving dynamic content, as well as a framework for data extraction. It specifically covers anti-bot bypass techniques, such as rotating proxies and spoofing headers, to avoid IP blocks and detection systems. The capability surface extend

Nexus-Mod-Manager is a desktop application and mod installation tool that connects to a central repository to download, install, organize, and update video game modifications. It serves as a centralized manager for handling game content directly from a desktop interface. The application includes dedicated download management capabilities that support pausing and resuming active file transfers to handle network bandwidth and large files. Users can also clean up partially installed or corrupted files automatically when an installation process fails or is interrupted. For configuration manageme

Masuit.Tools is a comprehensive static utility library for .NET and ASP.NET Core development. It provides a broad collection of reusable helper methods and infrastructure components that cover common programming tasks without requiring dependency injection or instance management. The library is organized as flat utility classes, making its functionality directly accessible from anywhere in a project. The toolkit distinguishes itself through a wide range of integrated capabilities that go beyond typical utility libraries. It includes a multithreaded range-request file downloader with pause and