How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
This project is a Python web scraping library and automated data collection suite. It provides tools for extracting structured data from websites, implementing web crawlers to navigate site links, and parsing HTML DOM structures to isolate specific elements and attributes. The toolkit includes a pipeline for processing unstructured text and cleaning raw web content to extract meaningful information. It also features capabilities for image data extraction and the integration of external APIs to retrieve structured data from remote endpoints. The system covers broad capability areas including
This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.
OnionScan is a free and open source tool for investigating the Dark Web.
reconftw is an attack surface management framework and reconnaissance workflow orchestrator designed to automate the discovery, mapping, and monitoring of external digital assets. It operates as a modular tool-chain pipeline that coordinates a sequence of security tools to perform intelligence gathering and vulnerability scanning. The project distinguishes itself through a cloud-native deployment model that parallelizes scanning workloads across a fleet of remote VPS instances to bypass local resource constraints. It utilizes container-based environment isolation to ensure consistent executio
TorBot is a suite of specialized tools for accessing and exploring hidden services on the Tor network. It functions as an onion crawler and web scraper that discovers hidden services by navigating onion domains to extract page titles, host names, and site descriptions.
The main features of dedsecinside/torbot are: TOR and Anonymity, Recursive Web Discovery, Onion Crawlers, Proxy-Enabled Scrapers, SOCKS5 Proxies, Web Crawling, Network Link Mappers, Network Relationship Visualizers.
Projects with overlapping indexed features include: remitchell/python-scraping — This project is a Python web scraping library and automated data collection suite. It provides tools for extracting… rchipka/node-osmosis — This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow… s-rah/onionscan — OnionScan is a free and open source tool for investigating the Dark Web. six2dez/reconftw — reconftw is an attack surface management framework and reconnaissance workflow orchestrator designed to automate the… megadose/onionsearch — OnionSearch is a script that scrapes urls on different .onion search engines. awesome-selfhosted/awesome-selfhosted — This project is a community-curated directory of open-source software designed for deployment in private server…