How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
Creeper is a next-generation crawler which fetches web page by creeper script. As a cross-platform embedded crawler, you can use it for your news app, subscribe program, etc.
The main features of wspl/creeper are: Go Crawling Frameworks, Web Crawlers.
Open-source alternatives to wspl/creeper include: puerkitobio/gocrawl — gocrawl is a polite, slim and concurrent web crawler written in Go. montferret/ferret. henrylee2cn/pholcus — Pholcus is a distributed web crawler framework written in Go designed for high-concurrency data extraction. It… asciimoo/colly — Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages,… gocolly/colly — Colly is a high-performance web scraping framework designed for the automated extraction of structured data from… geziyor/geziyor — Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
gocrawl is a polite, slim and concurrent web crawler written in Go.
Pholcus is a distributed web crawler framework written in Go designed for high-concurrency data extraction. It functions as a distributed crawling orchestrator and dynamic data extraction engine, utilizing a server-client architecture to coordinate tasks across multiple nodes. The system integrates a headless browser engine to render dynamic content and execute JavaScript, allowing it to extract data from single-page applications. It features a web-based management interface for configuring spider parameters and monitoring execution progress, alongside the ability to update extraction rules v
Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s