henrylee2cn/pholcus
Pholcus
Pholcus is a distributed web crawler framework written in Go designed for high-concurrency data extraction. It functions as a distributed crawling orchestrator and dynamic data extraction engine, utilizing a server-client architecture to coordinate tasks across multiple nodes.
The system integrates a headless browser engine to render dynamic content and execute JavaScript, allowing it to extract data from single-page applications. It features a web-based management interface for configuring spider parameters and monitoring execution progress, alongside the ability to update extraction rules via hot-reloading configuration files without restarting the system.
Traffic management is handled through proxy pool rotation and request randomization to evade bot detection and avoid rate limits. The framework includes state-based checkpoint recovery to resume tasks after failures and provides pluggable storage adapters to export extracted data to databases, message queues, or files.
Features
- Scraping Architectures - Provides a distributed architecture for scaling high-concurrency data extraction across multiple server and client nodes.
- Structured Data Extraction - Provides a dynamic engine for defining scraping logic to extract structured data from websites.
- Web Data Collection - Enables the creation of custom spiders with specific extraction rules to gather and save web-based information.
- Distributed Crawl Coordination - Coordinates high-concurrency crawling tasks across multiple nodes using a central server-client distribution model.
- Outbound IP Rotation - Cycles outgoing requests through a pool of proxy IP addresses to avoid rate limits and IP bans.
- Proxy and User-Agent Rotation Middleware - Features middleware that rotates proxy IP addresses and user agents to evade bot detection and rate limits.
- Bot Detection Bypass - Bypasses bot detection and rate limits by rotating proxy IPs and mimicking human browser behavior.
- Anti-Bot Evasion - Implements request randomization and cookie/user-agent management to mimic human behavior and bypass bot detection.
- JavaScript-Rendered Content Extractors - Executes JavaScript via a headless browser engine to extract structured data from dynamic single-page applications.
- Distributed Crawler Orchestrators - A distributed web scraping framework written in Go for high-concurrency data extraction across multiple nodes.
- Headless Crawlers - Utilizes a headless browser to render dynamic content and execute JavaScript to bypass security checks.
- Dynamic Content Extraction - Fetches data from single-page applications by rendering JavaScript-heavy content on the server.
- Web Crawling Orchestrators - Functions as a coordinator managing proxy rotation and checkpointing for reliable large-scale data collection.
- Extraction Rule Sets - Implements scraping logic through compiled code or hot-reloaded configuration files for flexible data extraction.
- Crawl State Recovery - Persists task status and request progress to resume crawling from the last successful checkpoint after a failure.
- Configuration Hot-Reloading - Allows scraping rules and extraction patterns to be updated in real time without restarting the system.
- Headless Browser Orchestrators - Implements a system to manage and control headless browser instances for rendering dynamic single-page applications.
- Crawler Configuration Managers - Provides a visual interface to control high-concurrency scraping tasks and monitor execution progress.
- Management Interfaces - Provides a web-based interface for configuring spider parameters, managing task execution, and monitoring scraping progress.
- Web Scraping Management Interfaces - Ships a web-based management interface for configuring spider parameters and monitoring distributed scraping processes.
- Go Crawling Frameworks - Distributed, high-concurrency web crawler.
- Web Crawlers - Distributed high-concurrency crawler.
