s0md3v/Photon
Photon
Photon is a command-line web crawler designed for security reconnaissance and information gathering. It systematically traverses websites to discover URLs, map domain infrastructure, and identify associated subdomains by retrieving DNS records.
The tool distinguishes itself through its ability to perform deep content analysis, including the extraction of sensitive data such as API keys and authentication tokens using user-defined regular expressions. It supports offline inspection by cloning crawled web content to the local filesystem, allowing for structural analysis without additional network activity.
The crawler utilizes multi-threaded execution to maximize throughput during discovery and supports proxy-aware routing to manage traffic origin. Its architecture is built for integration into automated security workflows, allowing users to pipe discovered metadata and extracted patterns directly to standard output or export results into structured files for further processing.
Features
- Infrastructure Reconnaissance - Traverses websites to discover URLs, extract sensitive data, and map domain infrastructure for security analysis.
- Security Crawlers - Provides a scriptable command-line crawler that saves content locally and pipes discovered metadata into security workflows.
- Security - Integrates web crawling and data extraction tasks into automated shell scripts and security pipelines.
- Subdomain Enumeration Tools - Identifies associated subdomains and retrieves DNS records to expand the scope of web reconnaissance.
- Web Crawling - Systematically traverses websites to discover URLs and metadata by following links.
- Sensitive Data Scanners - Scans crawled web content for exposed API keys, authentication tokens, and other sensitive secrets.
- Security Assessment Frameworks - Gathers structured website data and metadata to support security auditing and vulnerability assessment workflows.
- Access Key Management - Scans crawled content for high-entropy strings to detect exposed authentication tokens and API keys.
- Secret Extractors - Applies custom regular expressions to identify exposed API keys and secrets during web crawls.
- Reconnaissance and Discovery - Incredibly fast crawler designed for OSINT.
- Web Crawling and Scraping - Crawler specifically designed for OSINT data extraction.
- Network and Web Reconnaissance - Fast crawler for extracting URLs, emails, and files.
- Open Source Intelligence - Web crawler for extracting URLs, emails, and secrets.
- Reconnaissance and OSINT - Fast crawler designed for OSINT data collection.
- Security Tools - Incredibly fast crawler designed for OSINT.
- Web Reconnaissance - Fast web crawler for information gathering.
- Recursive Structure Traversers - Maintains a dynamic queue of discovered URLs to systematically navigate website structures.
- Domain Configuration Tools - Resolves and saves subdomain information to map domain infrastructure and DNS configuration.
- Pattern Extraction Utilities - Applies user-defined regular expressions during the crawl process to capture specific strings of interest.
- Web Content Scrapers - Clones crawled web content to the local filesystem for offline analysis and inspection.
- Proxy-Aware Network Clients - Routes outgoing HTTP requests through configurable external proxy servers to mask traffic origin.
- Traffic Proxying - Routes web requests through third-party services to distribute traffic origin during reconnaissance.
- Local Filesystem Storage - Saves crawled web pages and metadata to the local disk for offline inspection and structural analysis.
- Discovery Result Exporters - Saves discovered URLs and extracted data into structured files for further analysis.
- Offline Analysis Engines - Clones web content to the local filesystem to enable offline structural analysis and inspection of target websites.
- Data Extraction - Applies user-defined pattern matching to identify and capture specific strings during traversal.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
