freeok/so-novel
So Novel
so-novel is a web novel downloader and scraping engine designed to extract structured text from websites and convert it into electronic book formats. It functions as a multi-interface content extractor, providing a shared backend accessible via a web-based management dashboard, a terminal user interface, and a command line interface.
The system utilizes a rule-driven approach for data extraction, using CSS selectors and XPath rules defined in external configuration files to map web elements to specific data fields. To maintain access to content, it includes a proxy-routed request pipeline to bypass regional restrictions and anti-scraping protections.
The project converts raw extracted text into structured electronic formats such as EPUB, PDF, TXT, and HTML. It also includes utilities for translating content between Simplified and Traditional Chinese encoding standards and supports simultaneous searching across multiple aggregated web sources.
Deployment options include containerized images, shell scripts, and the ability to compile the application into a standalone native binary.
Features
- Web Scraping - Uses customizable CSS selectors and XPath rules to extract structured content from diverse web pages.
- Web Scraping - Extracts structured information from websites using customizable rule files based on CSS selectors, XPath, and regular expressions.
- Artwork and Novel Downloaders - Downloads online novels and chapters from various websites for offline archival and reading.
- Web Content Extractors - Transforms unstructured web pages into structured data formats based on predefined extraction rules.
- Document Format Converters - Converts raw extracted text into structured electronic formats including EPUB, PDF, and TXT.
- Document Format Conversions - Transforms extracted web text into structured electronic document formats such as EPUB, PDF, and TXT.
- eBook Compilers - Assembles scraped web content into standardized eBook formats including EPUB, MOBI, and PDF.
- Multi-Format Exporters - Converts extracted web content into standardized formats including EPUB, PDF, TXT, and HTML with optimized layouts.
- Web Element Mappings - Supports mapping web elements using CSS selectors or XPaths in configuration files to identify target content.
- Document Format Transformations - Transforms extracted raw web data into structured electronic document formats for storage and reading.
- Pattern-Based Extraction - Defines and switches between data extraction patterns via configuration files to support diverse sources.
- Proxy Routing - Routes outbound network traffic through configurable proxies to bypass regional restrictions and anti-scraping protections.
- Extraction Rule Sets - Manages serialized logic used to parse and extract data from various target websites.
- Web Element Mapping Rules - Maps website elements to specific data fields by switching between predefined configuration sets for different sources.
- Custom Extraction Rule Definitions - Allows the creation and loading of site-specific rules that define how content is parsed from different websites.
- CSS Selector Data Extractors - Parses unstructured web pages using configurable CSS selectors and XPath rules defined in external files.
- Web Scraping Engines - Implements a high-performance engine for automated extraction of structured text using CSS and XPath.
- Manga Chapter Downloaders - High-performance downloader for fetching multiple novel chapters simultaneously for offline reading.
- System Command Dispatchers - Implements a shared backend that routes commands across web, terminal, and command line interfaces.
- Network Proxy Configurations - Routes outbound network traffic through configurable proxy hosts to bypass regional access restrictions.
- Multi-threaded Downloading - Fetches multiple web resources simultaneously using a configurable concurrency model and retry logic.
- Anti-Bot Evasion - Implements techniques to bypass bot detection and security layers using proxy-routed requests.
- Protection Bypassers - Circumvents anti-scraping security layers and bot protections by routing requests through proxy services.
- Web Management Dashboards - Provides a browser-based dashboard for managing data extraction tasks and system configurations.
- Multi-Interface Access - Offers a shared backend accessible via a web-based dashboard, a terminal user interface, and a command line interface.
- Offline Web Page Archivers - Captures web-based novels and saves them as local files for offline reading on electronic devices.
- Website Crawlers and Scrapers - Functions as a deployment-ready system for recursively retrieving and structuring web novel content.
