digininja/CeWL
CeWL
CeWL is a custom wordlist generator and web crawling security tool designed to extract unique words and metadata from websites. It functions as an OSINT metadata extractor and security scanner, identifying potential passwords and usernames by analyzing HTML and JavaScript content.
The tool differentiates itself by combining recursive spidering with metadata extraction, allowing it to collect email addresses, author names, and creator metadata from web pages and linked files. It also captures domains, subdomains, and path components to include in generated lists.
Broad capabilities include web application spidering with depth control and regular expression filtering, as well as network request management using custom headers and proxy authentication. The system supports accessing restricted sites via Basic or Digest authentication and provides data processing utilities for word frequency analysis and list formatting.
The project is available as a containerized security scanner, packaged as a portable image to eliminate manual environment setup.
Features
- Web Spiders - Implements a recursive web spider that traverses links to a specified depth to harvest content.
- Custom Wordlist Generation - Spiders websites to extract unique words of a minimum length from HTML and JavaScript for use in password attacks.
- Information Gathering - Performs reconnaissance by collecting emails and author names from website metadata.
- Entity Extraction - Extracts email addresses and author names from mailto links and file properties for reconnaissance.
- HTML Parsing and Extraction - Parses HTML and JavaScript to extract unique strings and words based on defined length criteria.
- Email and Identity Extraction - Extracts email addresses and author names from web pages and linked files to build username lists.
- Web Metadata Extractors - Extracts email addresses and author names from web responses and markup for OSINT purposes.
- Security Crawlers - Analyzes HTML and JavaScript content to discover potential usernames and passwords.
- Domain Structure Analyzers - Extracts domains, subdomains, and path components to analyze and catalog the hierarchical structure of a target website.
- Crawl Boundary Controls - Uses regular expressions to define inclusion and exclusion rules to restrict automated traversal to specific domains or paths.
- Proxy-Aware Network Clients - Supports routing network traffic through proxies with custom authentication to bypass restrictions.
- Brute Force Attack Preparation - Prepares tailored wordlists from target domains to increase the effectiveness of dictionary attacks.
- Containerized Scanners - Provides a portable Docker image for performing website security analysis.
- Security Testing and Auditing - Supports security auditing by creating potential credential lists based on a company's public web presence.
- Word Frequency Counters - Tallies word occurrences during crawling to prioritize the most frequent terms in generated lists.
- Password and Hash Tools - Custom wordlist generator based on website content.
- Wordlists and Enumeration - Custom wordlist generator based on website content.
