hakluke/hakrawler
Hakrawler
Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment.
The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter prevents overly deep traversal, and an in-memory visited-URL set eliminates duplicate work. Pending URLs are managed in a FIFO queue for systematic breadth-first coverage, and all discovered endpoints are streamed line by line to standard output, allowing straightforward piping to other Unix tools.
Hakrawler is purpose-built for web endpoint and script discovery, making it a practical component in information-gathering workflows during security assessments. Its lightweight command-line interface requires no configuration beyond a target URL.
