hakluke/hakrawler
Hakrawler
Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment.
The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter prevents overly deep traversal, and an in-memory visited-URL set eliminates duplicate work. Pending URLs are managed in a FIFO queue for systematic breadth-first coverage, and all discovered endpoints are streamed line by line to standard output, allowing straightforward piping to other Unix tools.
Hakrawler is purpose-built for web endpoint and script discovery, making it a practical component in information-gathering workflows during security assessments. Its lightweight command-line interface requires no configuration beyond a target URL.
Features
- Web Spiders - Parses HTML responses to extract hyperlinks and script tags, outputting a list of discovered URLs and JavaScript files.
- JavaScript Endpoint Extractors - Extracts JavaScript file locations from web pages to find potential endpoints or hidden functionality.
- Site URL Mappers - Gathers every link from a site to build a complete map of its pages and resources for security analysis.
- Web Crawling Reconnaissance - Crawls target websites to collect URLs and scripts as part of pre-exploitation reconnaissance for penetration testing.
- Web Enumeration - Enumerates web assets such as endpoints, scripts, and forms for asset inventory or vulnerability assessment.
- URL Crawl Queues - Manages pending URLs in a FIFO queue to enforce breadth-first discovery and systematic coverage of the target site.
- Visited URL Sets - Uses an in-memory hash set of visited URLs to prevent re-crawling the same resource and avoid redundant work.
- Crawl Depth Limiters - Implements a per-path depth counter that halts recursion at a user-defined maximum depth to prune overly deep branches.
- Security Reconnaissance Tools - A command-line utility that spiders a target domain to gather endpoints and script sources for penetration testing.
- Link Extractors - Parses HTML with CSS selectors to extract anchor href and script src references for further crawling.
- Web Script Discoverers - Crawls a website's pages to collect all URLs and JavaScript file locations for security analysis.
- Concurrent Request Pooling - Ships a fixed-size goroutine pool that fetches pages in parallel, throttling concurrency for balanced speed and politeness.
- Security Crawlers - Discovers all URLs and JavaScript files on a website to map attack surface during security testing.
- Script Source Locators - Scans website pages specifically to identify and list all linked JavaScript file locations for security analysis.
- Content Discovery - Simple web crawler for endpoint and asset discovery.
- Domain and IP Investigation - Discovers endpoints and assets on target domains.
- Go Crawling Frameworks - Fast crawler for endpoint and asset discovery.
- HTTP Probing and Crawling - Simple web crawler for discovering endpoints and assets.
- Reconnaissance Tools - Fast web crawler for endpoint and asset discovery.
- Web Asset Discovery - Fast web crawler for gathering URLs and JavaScript files.
- Reconnaissance and Dorking - Rapidly discover links and endpoints within web applications.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
