henrylee2cn/pholcus
Pholcus
Pholcus ist ein in Go geschriebenes Framework für verteilte Web-Crawler, das für die Datenextraktion mit hoher Nebenläufigkeit entwickelt wurde. Es fungiert als Orchestrator für verteiltes Crawling und als Engine für dynamische Datenextraktion und nutzt eine Server-Client-Architektur, um Aufgaben über mehrere Knoten hinweg zu koordinieren.
Das System integriert eine Headless-Browser-Engine, um dynamische Inhalte zu rendern und JavaScript auszuführen, wodurch Daten aus Single-Page-Anwendungen extrahiert werden können. Es bietet ein webbasiertes Management-Interface zur Konfiguration von Spider-Parametern und zur Überwachung des Ausführungsfortschritts sowie die Möglichkeit, Extraktionsregeln über Hot-Reloading-Konfigurationsdateien zu aktualisieren, ohne das System neu zu starten.
Das Traffic-Management erfolgt durch Proxy-Pool-Rotation und Request-Randomisierung, um Bot-Erkennung zu umgehen und Ratenbegrenzungen zu vermeiden. Das Framework umfasst eine zustandsbasierte Checkpoint-Wiederherstellung, um Aufgaben nach Fehlern fortzusetzen, und bietet steckbare Speicheradapter, um extrahierte Daten in Datenbanken, Nachrichtenwarteschlangen oder Dateien zu exportieren.
Features
- Scraping Architectures - Provides a distributed architecture for scaling high-concurrency data extraction across multiple server and client nodes.
- Structured Data Extraction - Provides a dynamic engine for defining scraping logic to extract structured data from websites.
- Web Data Collection - Enables the creation of custom spiders with specific extraction rules to gather and save web-based information.
- Distributed Crawl Coordination - Coordinates high-concurrency crawling tasks across multiple nodes using a central server-client distribution model.
- Outbound IP Rotation - Cycles outgoing requests through a pool of proxy IP addresses to avoid rate limits and IP bans.
- Proxy and User-Agent Rotation Middleware - Features middleware that rotates proxy IP addresses and user agents to evade bot detection and rate limits.
- Bot Detection Bypass - Bypasses bot detection and rate limits by rotating proxy IPs and mimicking human browser behavior.
- Anti-Bot Evasion - Implements request randomization and cookie/user-agent management to mimic human behavior and bypass bot detection.
- JavaScript-Rendered Content Extractors - Executes JavaScript via a headless browser engine to extract structured data from dynamic single-page applications.
- Distributed Crawler Orchestrators - A distributed web scraping framework written in Go for high-concurrency data extraction across multiple nodes.
- Headless Crawlers - Utilizes a headless browser to render dynamic content and execute JavaScript to bypass security checks.
- Dynamic Content Extraction - Fetches data from single-page applications by rendering JavaScript-heavy content on the server.
- Web Crawling Orchestrators - Functions as a coordinator managing proxy rotation and checkpointing for reliable large-scale data collection.
- Extraction Rule Sets - Implements scraping logic through compiled code or hot-reloaded configuration files for flexible data extraction.
- Crawl State Recovery - Persists task status and request progress to resume crawling from the last successful checkpoint after a failure.
- Configuration Hot-Reloading - Allows scraping rules and extraction patterns to be updated in real time without restarting the system.
- Headless Browser Orchestrators - Implements a system to manage and control headless browser instances for rendering dynamic single-page applications.
- Crawler Configuration Managers - Provides a visual interface to control high-concurrency scraping tasks and monitor execution progress.
- Management Interfaces - Provides a web-based interface for configuring spider parameters, managing task execution, and monitoring scraping progress.
- Web Scraping Management Interfaces - Ships a web-based management interface for configuring spider parameters and monitoring distributed scraping processes.
- Go Crawling Frameworks - Distributed, high-concurrency web crawler.
- Web Crawlers - Distributed high-concurrency crawler.
