Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
Subfinder is a security reconnaissance framework designed for subdomain enumeration and attack surface management. It functions as a discovery engine that identifies and maps internet-exposed infrastructure, cloud-hosted assets, and network ranges to maintain a comprehensive inventory of an organization's digital footprint. The project distinguishes itself through a modular, template-driven scanning engine that executes security checks against discovered assets. It leverages cloud-native asset discovery to query provider APIs and infrastructure metadata, while supporting distributed agent orc
Gobuster is a command-line security utility designed for brute-force discovery of hidden infrastructure and content. It operates by systematically testing wordlists against target network services to identify files, directories, subdomains, and cloud storage buckets. The tool utilizes a concurrent worker pool to execute these requests in parallel, ensuring efficient scanning across various network environments. The project distinguishes itself through a modular plugin architecture that supports multiple discovery modes, including HTTP, DNS, and TFTP. This design allows for protocol-agnostic r
Feroxbuster is an HTTP directory brute forcer and web resource enumerator designed to discover hidden files and directories on web servers. It functions as a recursive URL scanner that identifies unlinked endpoints and API resources by combining wordlist-based scanning with automated crawling. The tool operates as a proxy-aware fuzzer, allowing network requests to be routed through HTTP or SOCKS proxies for traffic interception or anonymity. It utilizes recursive directory crawling to automatically queue discovered paths and find nested content. The system includes capabilities for discovery