How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve
Pholcus is a distributed web crawler framework written in Go designed for high-concurrency data extraction. It functions as a distributed crawling orchestrator and dynamic data extraction engine, utilizing a server-client architecture to coordinate tasks across multiple nodes. The system integrates a headless browser engine to render dynamic content and execute JavaScript, allowing it to extract data from single-page applications. It features a web-based management interface for configuring spider parameters and monitoring execution progress, alongside the ability to update extraction rules v
Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s
A simple, higher level interface for Go web scraping.
The main features of yhat/scrape are: Go Crawling Frameworks.
Projects with overlapping indexed features include: asciimoo/colly — Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages,… hakluke/hakrawler — Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and… henrylee2cn/pholcus — Pholcus is a distributed web crawler framework written in Go designed for high-concurrency data extraction. It… montferret/ferret. puerkitobio/fetchbot — Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies and crawl delays. puerkitobio/gocrawl — gocrawl is a polite, slim and concurrent web crawler written in Go.