This project is a Python web scraping library and automated data collection suite. It provides tools for extracting structured data from websites, implementing web crawlers to navigate site links, and parsing HTML DOM structures to isolate specific elements and attributes. The toolkit includes a pipeline for processing unstructured text and cleaning raw web content to extract meaningful information. It also features capabilities for image data extraction and the integration of external APIs to retrieve structured data from remote endpoints. The system covers broad capability areas including
This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.
OnionScan is a free and open source tool for investigating the Dark Web.
reconftw is an attack surface management framework and reconnaissance workflow orchestrator designed to automate the discovery, mapping, and monitoring of external digital assets. It operates as a modular tool-chain pipeline that coordinates a sequence of security tools to perform intelligence gathering and vulnerability scanning. The project distinguishes itself through a cloud-native deployment model that parallelizes scanning workloads across a fleet of remote VPS instances to bypass local resource constraints. It utilizes container-based environment isolation to ensure consistent executio
TorBot ist eine Suite spezialisierter Tools für den Zugriff auf und die Erkundung von Hidden Services im Tor-Netzwerk. Es fungiert als Onion-Crawler und Web-Scraper, der Hidden Services durch das Navigieren in Onion-Domains entdeckt, um Seitentitel, Hostnamen und Seitenbeschreibungen zu extrahieren.
Die Hauptfunktionen von dedsecinside/torbot sind: TOR and Anonymity, Recursive Web Discovery, Onion Crawlers, Proxy-Enabled Scrapers, SOCKS5 Proxies, Web Crawling, Network Link Mappers, Network Relationship Visualizers.
Open-Source-Alternativen zu dedsecinside/torbot sind unter anderem: remitchell/python-scraping — This project is a Python web scraping library and automated data collection suite. It provides tools for extracting… rchipka/node-osmosis — This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow… s-rah/onionscan — OnionScan is a free and open source tool for investigating the Dark Web. six2dez/reconftw — reconftw is an attack surface management framework and reconnaissance workflow orchestrator designed to automate the… megadose/onionsearch — OnionSearch is a script that scrapes urls on different .onion search engines. awesome-selfhosted/awesome-selfhosted — This project is a community-curated directory of open-source software designed for deployment in private server…