henrylee2cn/pholcus
Pholcus
Pholcus est un framework de web crawler distribué écrit en Go, conçu pour l'extraction de données à haute concurrence. Il fonctionne comme un orchestrateur de crawling distribué et un moteur d'extraction de données dynamique, utilisant une architecture serveur-client pour coordonner les tâches sur plusieurs nœuds.
Le système intègre un moteur de navigateur headless pour rendre le contenu dynamique et exécuter du JavaScript, lui permettant d'extraire des données d'applications monopages (SPA). Il dispose d'une interface de gestion web pour configurer les paramètres des spiders et surveiller la progression de l'exécution, ainsi que la capacité de mettre à jour les règles d'extraction via des fichiers de configuration à chaud (hot-reloading) sans redémarrer le système.
La gestion du trafic est assurée par la rotation de pools de proxys et la randomisation des requêtes pour échapper à la détection des bots et éviter les limites de débit. Le framework inclut une récupération de points de contrôle basée sur l'état pour reprendre les tâches après des échecs et fournit des adaptateurs de stockage enfichables pour exporter les données extraites vers des bases de données, des files d'attente de messages ou des fichiers.
Features
- Scraping Architectures - Provides a distributed architecture for scaling high-concurrency data extraction across multiple server and client nodes.
- Structured Data Extraction - Provides a dynamic engine for defining scraping logic to extract structured data from websites.
- Web Data Collection - Enables the creation of custom spiders with specific extraction rules to gather and save web-based information.
- Distributed Crawl Coordination - Coordinates high-concurrency crawling tasks across multiple nodes using a central server-client distribution model.
- Outbound IP Rotation - Cycles outgoing requests through a pool of proxy IP addresses to avoid rate limits and IP bans.
- Proxy and User-Agent Rotation Middleware - Features middleware that rotates proxy IP addresses and user agents to evade bot detection and rate limits.
- Bot Detection Bypass - Bypasses bot detection and rate limits by rotating proxy IPs and mimicking human browser behavior.
- Anti-Bot Evasion - Implements request randomization and cookie/user-agent management to mimic human behavior and bypass bot detection.
- JavaScript-Rendered Content Extractors - Executes JavaScript via a headless browser engine to extract structured data from dynamic single-page applications.
- Distributed Crawler Orchestrators - A distributed web scraping framework written in Go for high-concurrency data extraction across multiple nodes.
- Headless Crawlers - Utilizes a headless browser to render dynamic content and execute JavaScript to bypass security checks.
- Dynamic Content Extraction - Fetches data from single-page applications by rendering JavaScript-heavy content on the server.
- Web Crawling Orchestrators - Functions as a coordinator managing proxy rotation and checkpointing for reliable large-scale data collection.
- Extraction Rule Sets - Implements scraping logic through compiled code or hot-reloaded configuration files for flexible data extraction.
- Crawl State Recovery - Persists task status and request progress to resume crawling from the last successful checkpoint after a failure.
- Configuration Hot-Reloading - Allows scraping rules and extraction patterns to be updated in real time without restarting the system.
- Headless Browser Orchestrators - Implements a system to manage and control headless browser instances for rendering dynamic single-page applications.
- Crawler Configuration Managers - Provides a visual interface to control high-concurrency scraping tasks and monitor execution progress.
- Management Interfaces - Provides a web-based interface for configuring spider parameters, managing task execution, and monitoring scraping progress.
- Web Scraping Management Interfaces - Ships a web-based management interface for configuring spider parameters and monitoring distributed scraping processes.
- Go Crawling Frameworks - Distributed, high-concurrency web crawler.
- Web Crawlers - Distributed high-concurrency crawler.
