6 repository-uri
Awesome GitHub RepositoriesHTTP Probing and Crawling
Tools for identifying live web services and spidering site content.
Explore 6 awesome GitHub repositories matching part of an awesome list · HTTP Probing and Crawling. Refine with filters or upvote what's useful.
Awesome HTTP Probing and Crawling GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- projectdiscovery/katana
projectdiscovery/katana
15,584Vezi pe GitHubKatana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a utility for identifying endpoints, forms, and API structures across web targets by combining standard HTTP request traversal with headless browser automation to render dynamic, JavaScript-heavy content. The tool distinguishes itself through its ability to maintain authenticated sessions and handle complex web interactions, such as automated form submission and captcha resolution. It provides granular control over the discovery process, allowing users to define specific crawl
Next-generation crawling and spidering framework.
Goclicrawlergocrawler - projectdiscovery/httpx
projectdiscovery/httpx
10,053Vezi pe GitHubhttpx is a suite of tools and libraries for HTTP reconnaissance, infrastructure discovery, and DNS resolution. It functions as a command line toolkit for extracting metadata and status codes from HTTP targets and CIDR ranges, as well as a Go library for integrating these probing capabilities into custom programs. The project distinguishes itself through specialized infrastructure profiling, using TLS fingerprinting to extract JARM hashes and certificate details. It identifies underlying components such as CDN usage, Autonomous System Numbers, and CNAMEs to map web server software and infrastr
Fast and multi-purpose HTTP toolkit for probing and analysis.
Go - hakluke/hakrawler
hakluke/hakrawler
4,993Vezi pe GitHubHakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve
Simple web crawler for discovering endpoints and assets.
Gobugbountycrawlinghacking - edoardottt/cariddi
edoardottt/cariddi
3,427Vezi pe GitHubTake a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Crawler for finding endpoints, secrets, and potential vulnerabilities.
Go - tomnomnom/httprobe
tomnomnom/httprobe
3,112Vezi pe GitHubTake a list of domains and probe for working http and https servers.
Simple tool to check which domains have working HTTP/HTTPS servers.
Go - jaeles-project/gospider
jaeles-project/gospider
2,973Vezi pe GitHubGospider - Fast web spider written in Go
Fast web spider for gathering URLs and endpoints.
Go
