hect0x7/JMComic-Crawler-Python

Features

• Comic and Manga Downloaders - Provides a high-performance utility for batch downloading digital comic and manga chapters from hosting services.
• Obfuscated Image Reconstruction - Reconstructs images that have been sliced and shuffled by server-side obfuscation to restore them to a viewable state.
• Media Decryption - Implements cryptographic processes to decrypt protected images and reverse server-side pixel obfuscation.
• Keyword Search Crawlers - Queries the platform's search API by keywords and tags to collect matching manga content.
• Manga Chapter Downloaders - Downloads manga chapters to local storage for offline reading with automated organization.
• Manga Scrapers - Implements a high-performance scraper to normalize manga content and metadata from hosting services.
• Comic Archive Converters - Transforms retrieved comic images into PDF, ZIP, or long-strip image formats for archiving.
• Content Ranking Aggregators - Fetches popularity rankings and curated lists of comic content based on view counts.
• Local-Remote Media Synchronizers - Synchronizes online manga libraries with local storage by monitoring and downloading new chapters.
• Manga Metadata Retrieval - Fetches image links and descriptive metadata for manga titles via asynchronous requests.
• Scheduled Content Updates - Periodically checks remote sources for new chapters and automates content updates.
• Web Content Scraping - Automates the retrieval of images and metadata from comic websites for local storage.
• Web Media Scrapers - Uses asynchronous requests and multi-threading to efficiently extract media and metadata from the web.
• Asynchronous Content Retrievers - Uses asynchronous requests and generators to retrieve manga images and metadata at high speed.
• Iterator-Based Pagination - Navigates large datasets of comic albums through automated page-by-page retrieval using generators.
• Search and Indexing - Integrates search and indexing to retrieve album details, rankings, and personal collection metadata.
• HTML Parsing and Extraction - Extracts structured data, tags, and result counts from HTML search and category pages.
• Media Category Filters - Filters search results by predefined media categories and ranking metrics before retrieval.
• Dynamic Domain Resolution - Programmatically resolves current web addresses from permanent URLs to bypass frequent domain changes.
• Media Category Browsing - Provides navigation systems to explore comic content via categories and time-based rankings.
• Format Converters - Converts downloaded comic images between different formats, such as converting a series of images into a long PNG strip.
• Media Content Archivers - Batch downloads digital media from web sources and organizes them for offline use.
• Obfuscated Image Decryptions - Process encrypted image files into a viewable format to ensure content is accessible after downloading.
• Batch Image Downloads - Processes bulk retrieval of multiple albums and chapters in a single operation with fault tolerance.
• Media Content Retrievers - Fetches content programmatically using mobile API endpoints and automatic domain rotation.
• Domain Rotation Strategies - Uses dynamic domain rotation and proxy routing to circumvent network blocks and bot protections.
• Bot Detection Bypass - Mimics legitimate browser behavior and uses app-level encryption to evade bot detection systems.
• Protection Bypassers - Circumvents automated firewalls and manages domain switching to maintain stable connectivity.
• Asynchronous Pipelines - Implements an asynchronous pipeline that separates network I/O from decryption to maximize data throughput.
• Image Metadata Retrievers - Retrieves technical image metadata, including specific scramble IDs required for decryption.
• Dynamic Domain Rotation - Uses dynamic domain rotation and failure tracking to maintain connectivity and bypass network blocks.
• Manga Album Browsers - Allows searching for manga using filter criteria and retrieving album details, rankings, and category lists.
• High-Volume Web Scrapers - Implements a high-performance crawler optimized for large-scale collection and anti-blocking.
• Resource Detail Retrievals - Retrieves detailed metadata and information for specific manga albums or chapters using their unique IDs.
• Web Crawlers - Functions as a programmable system for asynchronously visiting URLs and extracting manga metadata.
• Content Sync Actions - Integrates with GitHub Actions to schedule automated content updates and library synchronization.
• Storage Path Templates - Generates local storage directories and filenames using dynamic metadata templates.
• Media Post-Processing Utilities - Transforms raw image bytes into PDF, ZIP, or long-strip formats after the download process completes.
• Incremental Downloaders - Provides functionality to identify and download only new chapters added to a comic since the last update.
• Content Formats and Exporting - Converts downloaded images into portable formats such as PDF, ZIP, or long-strip images.
• Automated File Organizers - Automatically organizes downloaded files into directory structures based on predefined naming rules and placeholders.
• CLI Execution - Offers a command-line interface for executing downloads and metadata lookups without a GUI.
• Custom Plugin Registrations - Implements a registration system for custom plugins to add specialized behaviors to the crawler.
• Python API Clients - Implements a native Python library for object-oriented interaction with the comic service API.
• Concurrency Controllers - Limits the number of simultaneous image and chapter requests to prevent resource exhaustion and server blocks.
• API Token Generators - Generates signed tokens from secrets and timestamps to authenticate API requests.
• Comic Format Transformations - Transforms sequences of comic images into reader-friendly formats like PDF and long-strip images.
• Custom Client Implementations - Allows users to replace the default HTTP networking layer with custom client implementations for specialized request handling.
• CI-Based - Executes content retrieval and exports using CI tools to bypass regional network restrictions.
• Multi-Interface Content Retrieval - Toggles between web and API interfaces to optimize request efficiency and circumvent IP restrictions.
• Network Proxy Configurations - Supports routing network requests through custom HTTP and HTTPS proxies to bypass connectivity restrictions.
• Bot-Protection Bypassing Downloader - Bypasses bot detection systems using proxy routing and dynamic domain rotation.
• Multi-threaded Downloading - Increases overall download speed by fetching content segments in parallel across multiple threads.
• Session Authentication - Handles authentication sessions and manages access to personal favorite collections.
• Browser Cookie Imports - Extracts session cookies directly from local web browsers to enable authenticated API access.
• Cookie-Based Authentication Bridges - Bridges browser cookies and session data into the crawler to access restricted content and bypass bot protection.
• User Account Management - Manages user credentials and authentication states to establish secure sessions.
• Asynchronous Request Pipelines - Implements an asynchronous pipeline that decouples network I/O from image decryption to maximize throughput.
• Event-Driven Plugin Logic - Runs custom logic automatically when specific events occur, such as downloading images or albums.
• Plugin Extenders - Supports augmenting core application functionality by loading custom external libraries at runtime.
• Core Capability Extensions - Allows extending the fundamental operational logic of the crawler through custom plugins and modules.
• Multi-threaded Data Extraction - Implements concurrency patterns to distribute web scraping tasks across multiple workers for faster data extraction.
• Hook-Based Plugin Systems - Provides a hook-based plugin system to execute custom logic at specific stages of the download lifecycle.
• Domain Connectivity Tests - Verifies if specific service domains are reachable from the current IP to manage domain rotation.
• Asynchronous API Clients - Provides an asynchronous Python interface for interacting with comic service web and mobile endpoints.
• Retry and Backoff Logic - Implements automated retry strategies and domain blacklisting to recover from network timeouts.
• Decryption - Decrypts encrypted API response payloads to retrieve plain JSON metadata.

Star history