henrylee2cn/pholcus
Pholcus
Pholcus es un framework de web crawler distribuido escrito en Go, diseñado para la extracción de datos de alta concurrencia. Funciona como un orquestador de rastreo distribuido y un motor de extracción de datos dinámicos, utilizando una arquitectura servidor-cliente para coordinar tareas a través de múltiples nodos.
El sistema integra un motor de navegador headless para renderizar contenido dinámico y ejecutar JavaScript, permitiéndole extraer datos de aplicaciones de una sola página (SPA). Cuenta con una interfaz de gestión basada en web para configurar parámetros de arañas y monitorear el progreso de la ejecución, junto con la capacidad de actualizar reglas de extracción mediante archivos de configuración con recarga en caliente (hot-reloading) sin reiniciar el sistema.
La gestión del tráfico se maneja mediante la rotación de pools de proxies y la aleatorización de solicitudes para evadir la detección de bots y evitar límites de tasa. El framework incluye recuperación de puntos de control basada en estado para reanudar tareas tras fallos y proporciona adaptadores de almacenamiento conectables para exportar datos extraídos a bases de datos, colas de mensajes o archivos.
Features
- Scraping Architectures - Provides a distributed architecture for scaling high-concurrency data extraction across multiple server and client nodes.
- Structured Data Extraction - Provides a dynamic engine for defining scraping logic to extract structured data from websites.
- Web Data Collection - Enables the creation of custom spiders with specific extraction rules to gather and save web-based information.
- Distributed Crawl Coordination - Coordinates high-concurrency crawling tasks across multiple nodes using a central server-client distribution model.
- Outbound IP Rotation - Cycles outgoing requests through a pool of proxy IP addresses to avoid rate limits and IP bans.
- Proxy and User-Agent Rotation Middleware - Features middleware that rotates proxy IP addresses and user agents to evade bot detection and rate limits.
- Bot Detection Bypass - Bypasses bot detection and rate limits by rotating proxy IPs and mimicking human browser behavior.
- Anti-Bot Evasion - Implements request randomization and cookie/user-agent management to mimic human behavior and bypass bot detection.
- JavaScript-Rendered Content Extractors - Executes JavaScript via a headless browser engine to extract structured data from dynamic single-page applications.
- Distributed Crawler Orchestrators - A distributed web scraping framework written in Go for high-concurrency data extraction across multiple nodes.
- Headless Crawlers - Utilizes a headless browser to render dynamic content and execute JavaScript to bypass security checks.
- Dynamic Content Extraction - Fetches data from single-page applications by rendering JavaScript-heavy content on the server.
- Web Crawling Orchestrators - Functions as a coordinator managing proxy rotation and checkpointing for reliable large-scale data collection.
- Extraction Rule Sets - Implements scraping logic through compiled code or hot-reloaded configuration files for flexible data extraction.
- Crawl State Recovery - Persists task status and request progress to resume crawling from the last successful checkpoint after a failure.
- Configuration Hot-Reloading - Allows scraping rules and extraction patterns to be updated in real time without restarting the system.
- Headless Browser Orchestrators - Implements a system to manage and control headless browser instances for rendering dynamic single-page applications.
- Crawler Configuration Managers - Provides a visual interface to control high-concurrency scraping tasks and monitor execution progress.
- Management Interfaces - Provides a web-based interface for configuring spider parameters, managing task execution, and monitoring scraping progress.
- Web Scraping Management Interfaces - Ships a web-based management interface for configuring spider parameters and monitoring distributed scraping processes.
- Go Crawling Frameworks - Distributed, high-concurrency web crawler.
- Web Crawlers - Distributed high-concurrency crawler.
