kanasimi/work_crawler

Features

• Digital Comic Archivers - Automates the collection of manga and webtoons from diverse platforms and organizes them into compressed image archives.
• Plugin-Based - Uses a modular plugin-based architecture with site-specific scripts to define parsing and extraction rules for diverse platforms.
• Automated Content Retrievers - Provides an automated system for collecting comics and novels via GUI, CLI, or API interfaces.
• Comic and Manga Downloaders - Scrapes digital comics and novels from multiple international websites for offline reading.
• Comic Site Extractors - Extracts images and metadata from web-based comic platforms to automate digital manga collection.
• Incremental Novel Downloaders - Provides resumable, batch downloading of novel chapters and images for offline storage.
• Cookie-Based Session Authentication for Downloads - Uses imported browser cookies to authenticate and download restricted or member-only content from media platforms.
• E-book Generators - Packages extracted novel text and images into standardized electronic book files compatible with reading software.
• Novel to EPUB Converters - Transforms long-form web novel text and images into standardized EPUB e-books.
• Novel to EPUB Pipelines - Implements an end-to-end pipeline that scrapes web novel content and converts it directly into EPUB format.
• Multi-Site Content Crawlers - Extracts images and text from multiple international websites using a modular system of site-specific scraping scripts.
• Web Media Scrapers - Extracts images and text from diverse media platforms using specialized site-specific scraping logic.
• Content Processing Pipelines - Implements a sequential workflow that fetches raw web content and processes it into structured archives and e-books.
• Web Novel Aggregators - Implements specialized crawlers to import literary content from remote web novel catalogs.
• Custom Scraping Logic - Implements an extensibility layer allowing users to define specific scraping rules for different websites via a crawler library.
• Session-Cookie Persistences - Persists and reuses browser session cookies to access restricted or member-only content across multiple runs.
• Script Conversion - Transforms text between simplified and traditional Chinese scripts to standardize the language of downloaded content.
• Chinese Script Normalizers - Standardizes Chinese text by normalizing and converting between simplified and traditional script variants.
• Multi-Site Title Discovery - Implements logic to search for specific titles across multiple disparate web platforms simultaneously.
• International Content Retrievers - Retrieves digital comics and novels from international web platforms across multiple languages.
• Sequence Archiving - Extracts image sequences from manga sites and packages them into compressed archives for local storage.
• Multi-Platform Title Search - Locates specific titles across multiple supported websites and initiates downloads through a single action.
• Web Content Scraping - Gathers digital books and comics from international websites by parsing HTML and network responses across different languages.
• Manga E-book Converters - Downloads web-based literary works and converts them into e-book formats optimized for e-readers.
• Resumable Download State Management - Tracks download checkpoints via local state files to allow resuming interrupted tasks from the last completed chapter.
• Task Management Command-Line Interfaces - Provides a command-line interface for managing and executing download tasks with custom options and proxy settings.
• Fetch Quality Optimizers - Ensures maximum image quality by fetching the highest available resolution and re-downloading corrupted assets.
• Asset Quality Optimizers - Verifies the integrity of downloaded media files and automatically re-fetches corrupted assets.
• Graphical Management Interfaces - Provides a graphical user interface for managing download configurations, themes, and multi-language settings.
• Download Management Systems - Provides a state-tracking system to record checkpoints and resume interrupted downloads from the last completed chapter.
• HTTP Session Simulations - Mimics authenticated browser sessions using cookies to access account-specific restricted content.
• Multi-Interface Architectures - Provides a decoupled architecture that allows the core downloading engine to be controlled via CLI, GUI, and API.
• Task Triggering Interfaces - Ships a unified interface to trigger and monitor the crawling and download process via graphical and command-line tools.

Star 历史