kanasimi/work_crawler
Work Crawler
This project is a web-based manga and novel downloader and multi-site web scraper designed to extract images and text from diverse media platforms. It functions as a digital media archiver and EPUB e-book generator, using a plugin-based crawler architecture with site-specific scripts to define how content is extracted from various international websites.
The system distinguishes itself through authenticated web crawling, using browser cookie simulation to access restricted or member-only content. It includes specialized capabilities for digital comic archiving, which organizes image sequences into compressed archives, and a conversion pipeline that packages web novel text and images into standardized EPUB files.
The software provides comprehensive content processing, including image integrity verification to ensure maximum quality and middleware for standardizing simplified and traditional Chinese scripts. It manages downloads via a state-tracking system that allows for task resumption, and offers multiple ways to initiate crawls through a graphical user interface, command-line interface, or API.
Features
- Digital Comic Archivers - Automates the collection of manga and webtoons from diverse platforms and organizes them into compressed image archives.
- Plugin-Based - Uses a modular plugin-based architecture with site-specific scripts to define parsing and extraction rules for diverse platforms.
- Automated Content Retrievers - Provides an automated system for collecting comics and novels via GUI, CLI, or API interfaces.
- Comic and Manga Downloaders - Scrapes digital comics and novels from multiple international websites for offline reading.
- Comic Site Extractors - Extracts images and metadata from web-based comic platforms to automate digital manga collection.
- Incremental Novel Downloaders - Provides resumable, batch downloading of novel chapters and images for offline storage.
- Cookie-Based Session Authentication for Downloads - Uses imported browser cookies to authenticate and download restricted or member-only content from media platforms.
- E-book Generators - Packages extracted novel text and images into standardized electronic book files compatible with reading software.
- Novel to EPUB Converters - Transforms long-form web novel text and images into standardized EPUB e-books.
- Novel to EPUB Pipelines - Implements an end-to-end pipeline that scrapes web novel content and converts it directly into EPUB format.
- Multi-Site Content Crawlers - Extracts images and text from multiple international websites using a modular system of site-specific scraping scripts.
- Web Media Scrapers - Extracts images and text from diverse media platforms using specialized site-specific scraping logic.
- Content Processing Pipelines - Implements a sequential workflow that fetches raw web content and processes it into structured archives and e-books.
- Web Novel Aggregators - Implements specialized crawlers to import literary content from remote web novel catalogs.
- Custom Scraping Logic - Implements an extensibility layer allowing users to define specific scraping rules for different websites via a crawler library.
- Session-Cookie Persistences - Persists and reuses browser session cookies to access restricted or member-only content across multiple runs.
- Script Conversion - Transforms text between simplified and traditional Chinese scripts to standardize the language of downloaded content.
- Chinese Script Normalizers - Standardizes Chinese text by normalizing and converting between simplified and traditional script variants.
- Multi-Site Title Discovery - Implements logic to search for specific titles across multiple disparate web platforms simultaneously.
- International Content Retrievers - Retrieves digital comics and novels from international web platforms across multiple languages.
- Sequence Archiving - Extracts image sequences from manga sites and packages them into compressed archives for local storage.
- Multi-Platform Title Search - Locates specific titles across multiple supported websites and initiates downloads through a single action.
- Web Content Scraping - Gathers digital books and comics from international websites by parsing HTML and network responses across different languages.
- Manga E-book Converters - Downloads web-based literary works and converts them into e-book formats optimized for e-readers.
- Resumable Download State Management - Tracks download checkpoints via local state files to allow resuming interrupted tasks from the last completed chapter.
- Task Management Command-Line Interfaces - Provides a command-line interface for managing and executing download tasks with custom options and proxy settings.
- Fetch Quality Optimizers - Ensures maximum image quality by fetching the highest available resolution and re-downloading corrupted assets.
- Asset Quality Optimizers - Verifies the integrity of downloaded media files and automatically re-fetches corrupted assets.
- Graphical Management Interfaces - Provides a graphical user interface for managing download configurations, themes, and multi-language settings.
- Download Management Systems - Provides a state-tracking system to record checkpoints and resume interrupted downloads from the last completed chapter.
- HTTP Session Simulations - Mimics authenticated browser sessions using cookies to access account-specific restricted content.
- Multi-Interface Architectures - Provides a decoupled architecture that allows the core downloading engine to be controlled via CLI, GUI, and API.
- Task Triggering Interfaces - Ships a unified interface to trigger and monitor the crawling and download process via graphical and command-line tools.
