2 Repos
Awesome GitHub RepositoriesCrawler Extensions
Modular components that extend web crawling functionality with custom request headers and automated behavior.
Distinguishing note: Focuses on modular extensions for crawlers rather than core scraping logic.
Explore 2 awesome GitHub repositories matching web development · Crawler Extensions. Refine with filters or upvote what's useful.
Awesome Crawler Extensions GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- gocolly/colly
gocolly/colly
25,101Auf GitHub ansehenColly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks. The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into sp
Provides pluggable extensions to automate user agent rotation and referrer management during web data collection.
Gocrawlercrawlingframework - python3webspider/proxypool
Python3WebSpider/ProxyPool
6,223Auf GitHub ansehenProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability. The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a p
Allows adding new proxy sources by writing a class that defines target URLs and a parse method to extract host and port.
Pythonflaskhttpproxy
