hect0x7/JMComic-Crawler-Python
JMComic Crawler Python
JMComic-Crawler-Python is a high-performance asynchronous web scraper and API client designed to programmatically retrieve images and metadata from a comic hosting service. It functions as a media archiving tool for batch downloading albums and chapters, automating the process of saving content to a local filesystem.
The project is distinguished by its ability to reverse server-side pixel obfuscation, using a decryption tool to reconstruct sliced and shuffled images. To maintain stable connectivity, it utilizes a network bypass utility featuring dynamic domain rotation and proxy routing to circumvent bot protections and network blocks.
The crawler provides extensive capabilities for content management, including the conversion of downloaded images into PDF, ZIP, or long-strip formats. It covers broad functional areas such as user account authentication via browser cookie imports, asynchronous content searching, and automated synchronization of new chapters. The system also supports extensibility through a plugin-based event system and custom HTTP client implementations.
Users can execute downloads directly via a command line interface or automate workflows using continuous integration platforms.
Features
- Comic and Manga Downloaders - Provides a high-performance utility for batch downloading digital comic and manga chapters from hosting services.
- Obfuscated Image Reconstruction - Reconstructs images that have been sliced and shuffled by server-side obfuscation to restore them to a viewable state.
- Media Decryption - Implements cryptographic processes to decrypt protected images and reverse server-side pixel obfuscation.
- Keyword Search Crawlers - Queries the platform's search API by keywords and tags to collect matching manga content.
- Manga Chapter Downloaders - Downloads manga chapters to local storage for offline reading with automated organization.
- Manga Scrapers - Implements a high-performance scraper to normalize manga content and metadata from hosting services.
- Comic Archive Converters - Transforms retrieved comic images into PDF, ZIP, or long-strip image formats for archiving.
- Content Ranking Aggregators - Fetches popularity rankings and curated lists of comic content based on view counts.
- Local-Remote Media Synchronizers - Synchronizes online manga libraries with local storage by monitoring and downloading new chapters.
- Manga Metadata Retrieval - Fetches image links and descriptive metadata for manga titles via asynchronous requests.
- Scheduled Content Updates - Periodically checks remote sources for new chapters and automates content updates.
- Web Content Scraping - Automates the retrieval of images and metadata from comic websites for local storage.
- Web Media Scrapers - Uses asynchronous requests and multi-threading to efficiently extract media and metadata from the web.
- Asynchronous Content Retrievers - Uses asynchronous requests and generators to retrieve manga images and metadata at high speed.
- Iterator-Based Pagination - Navigates large datasets of comic albums through automated page-by-page retrieval using generators.
- Search and Indexing - Integrates search and indexing to retrieve album details, rankings, and personal collection metadata.
- HTML Parsing and Extraction - Extracts structured data, tags, and result counts from HTML search and category pages.
- Media Category Filters - Filters search results by predefined media categories and ranking metrics before retrieval.
- Dynamic Domain Resolution - Programmatically resolves current web addresses from permanent URLs to bypass frequent domain changes.
- Media Category Browsing - Provides navigation systems to explore comic content via categories and time-based rankings.
- Format Converters - Converts downloaded comic images between different formats, such as converting a series of images into a long PNG strip.
- Media Content Archivers - Batch downloads digital media from web sources and organizes them for offline use.
- Obfuscated Image Decryptions - Process encrypted image files into a viewable format to ensure content is accessible after downloading.
- Batch Image Downloads - Processes bulk retrieval of multiple albums and chapters in a single operation with fault tolerance.
- Media Content Retrievers - Fetches content programmatically using mobile API endpoints and automatic domain rotation.
- Domain Rotation Strategies - Uses dynamic domain rotation and proxy routing to circumvent network blocks and bot protections.
- Bot Detection Bypass - Mimics legitimate browser behavior and uses app-level encryption to evade bot detection systems.
- Protection Bypassers - Circumvents automated firewalls and manages domain switching to maintain stable connectivity.
- Asynchronous Pipelines - Implements an asynchronous pipeline that separates network I/O from decryption to maximize data throughput.
- Image Metadata Retrievers - Retrieves technical image metadata, including specific scramble IDs required for decryption.
- Dynamic Domain Rotation - Uses dynamic domain rotation and failure tracking to maintain connectivity and bypass network blocks.
- Manga Album Browsers - Allows searching for manga using filter criteria and retrieving album details, rankings, and category lists.
- High-Volume Web Scrapers - Implements a high-performance crawler optimized for large-scale collection and anti-blocking.
- Resource Detail Retrievals - Retrieves detailed metadata and information for specific manga albums or chapters using their unique IDs.
- Web Crawlers - Functions as a programmable system for asynchronously visiting URLs and extracting manga metadata.
- Content Sync Actions - Integrates with GitHub Actions to schedule automated content updates and library synchronization.
- Storage Path Templates - Generates local storage directories and filenames using dynamic metadata templates.
- Media Post-Processing Utilities - Transforms raw image bytes into PDF, ZIP, or long-strip formats after the download process completes.
- Incremental Downloaders - Provides functionality to identify and download only new chapters added to a comic since the last update.
- Content Formats and Exporting - Converts downloaded images into portable formats such as PDF, ZIP, or long-strip images.
- Automated File Organizers - Automatically organizes downloaded files into directory structures based on predefined naming rules and placeholders.
- CLI Execution - Offers a command-line interface for executing downloads and metadata lookups without a GUI.
- Custom Plugin Registrations - Implements a registration system for custom plugins to add specialized behaviors to the crawler.
- Python API Clients - Implements a native Python library for object-oriented interaction with the comic service API.
- Concurrency Controllers - Limits the number of simultaneous image and chapter requests to prevent resource exhaustion and server blocks.
- API Token Generators - Generates signed tokens from secrets and timestamps to authenticate API requests.
- Comic Format Transformations - Transforms sequences of comic images into reader-friendly formats like PDF and long-strip images.
- Custom Client Implementations - Allows users to replace the default HTTP networking layer with custom client implementations for specialized request handling.
- CI-Based - Executes content retrieval and exports using CI tools to bypass regional network restrictions.
- Multi-Interface Content Retrieval - Toggles between web and API interfaces to optimize request efficiency and circumvent IP restrictions.
- Network Proxy Configurations - Supports routing network requests through custom HTTP and HTTPS proxies to bypass connectivity restrictions.
- Bot-Protection Bypassing Downloader - Bypasses bot detection systems using proxy routing and dynamic domain rotation.
- Multi-threaded Downloading - Increases overall download speed by fetching content segments in parallel across multiple threads.
- Session Authentication - Handles authentication sessions and manages access to personal favorite collections.
- Browser Cookie Imports - Extracts session cookies directly from local web browsers to enable authenticated API access.
- Cookie-Based Authentication Bridges - Bridges browser cookies and session data into the crawler to access restricted content and bypass bot protection.
- User Account Management - Manages user credentials and authentication states to establish secure sessions.
- Asynchronous Request Pipelines - Implements an asynchronous pipeline that decouples network I/O from image decryption to maximize throughput.
- Event-Driven Plugin Logic - Runs custom logic automatically when specific events occur, such as downloading images or albums.
- Plugin Extenders - Supports augmenting core application functionality by loading custom external libraries at runtime.
- Core Capability Extensions - Allows extending the fundamental operational logic of the crawler through custom plugins and modules.
- Multi-threaded Data Extraction - Implements concurrency patterns to distribute web scraping tasks across multiple workers for faster data extraction.
- Hook-Based Plugin Systems - Provides a hook-based plugin system to execute custom logic at specific stages of the download lifecycle.
- Domain Connectivity Tests - Verifies if specific service domains are reachable from the current IP to manage domain rotation.
- Asynchronous API Clients - Provides an asynchronous Python interface for interacting with comic service web and mobile endpoints.
- Retry and Backoff Logic - Implements automated retry strategies and domain blacklisting to recover from network timeouts.
- Decryption - Decrypts encrypted API response payloads to retrieve plain JSON metadata.
