1 dépôt
Awesome GitHub RepositoriesCompiled Extraction Rules
Static compiled code used to optimize the speed and efficiency of data extraction processes.
Distinct from High-Performance Containers: Focuses on the execution logic of scraping rules rather than the data containers used to store the results.
Explore 1 awesome GitHub repository matching data & databases · Compiled Extraction Rules. Refine with filters or upvote what's useful.
Awesome Compiled Extraction Rules GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- andeya/pholcus
andeya/pholcus
7,578Voir sur GitHubPholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The
Uses static compiled code for high-performance scraping or dynamic files for hot-loading rules without restarting the system.
Go
