Awesome GitHub RepositoriesHTTP Probing and Crawling

Katana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a utility for identifying endpoints, forms, and API structures across web targets by combining standard HTTP request traversal with headless browser automation to render dynamic, JavaScript-heavy content. The tool distinguishes itself through its ability to maintain authenticated sessions and handle complex web interactions, such as automated form submission and captcha resolution. It provides granular control over the discovery process, allowing users to define specific crawl

httpx is a suite of tools and libraries for HTTP reconnaissance, infrastructure discovery, and DNS resolution. It functions as a command line toolkit for extracting metadata and status codes from HTTP targets and CIDR ranges, as well as a Go library for integrating these probing capabilities into custom programs. The project distinguishes itself through specialized infrastructure profiling, using TLS fingerprinting to extract JARM hashes and certificate details. It identifies underlying components such as CDN usage, Autonomous System Numbers, and CNAMEs to map web server software and infrastr

Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Take a list of domains and probe for working http and https servers.

Gospider - Fast web spider written in Go