yangyangwithgnu/hardseed
Hardseed
Hardseed is a command-line forum media scraper and content archiver. It extracts images and torrent seeds from internet forums in bulk, saving the media and associated metadata to local directories.
The tool utilizes a proxy-enabled web scraping engine that rotates network traffic through multiple proxy servers and protocols to bypass rate limits and network restrictions. It includes a keyword-based media filter that matches user-defined strings within entry titles to include or exclude specific topics.
The system manages data extraction through batch-processed thread iteration and range-limited topic processing. Downloaded assets are organized into a folder hierarchy using template-based directory mapping derived from category and timestamp metadata. All operations are managed via a terminal-based command interface.
Features
- Forum Media Scrapers - Extracts images and torrent seeds from internet forums in bulk to create local archives.
- Automated Media Archivers - Automates the collection and local storage of forum media and metadata.
- Command Line Interfaces - Provides a terminal-based interface for managing scraping and filtering routines.
- Title-Based Content Filtering - Filters forum topics by matching user-defined keywords within entry titles.
- Torrent Seed Scrapers - Scrapes and saves torrent files from targeted forum threads into organized directories.
- CLI Archivers - Saves forum media and metadata using a terminal application with structured naming conventions.
- Multi-Protocol Proxy Clients - Implements a client-side engine capable of routing traffic through multiple proxy protocols to bypass network restrictions.
- Proxy Rotation Services - Distributes network traffic across a rotating pool of proxy servers to circumvent rate limits.
- Proxy Routing - Routes automated download requests through multiple proxies to bypass network restrictions.
- Proxy-Enabled Media Fetchers - Routes data extraction traffic through proxy servers to bypass rate limits and restrictions.
- Torrent Seed Collectors - Automatically gathers torrent metadata and magnet links from forum topics.
- Index Range Selection - Constrains the scraping engine to a specific numerical range of forum IDs.
- Batch Processing Utilities - Processes forum entries in sequenced groups to extract media without overloading the source server.
- Batch Media Scraping - Extracts images and torrent seeds by iterating through forum threads in sequenced groups.
- Range-Limited Indexing - Provides the ability to target specific numerical ranges of forum IDs during the scraping process.
- Metadata-Driven Directory Mapping - Organizes downloaded assets into a folder hierarchy based on category and timestamp metadata.
