Awesome GitHub RepositoriesWeb Crawling and Scraping

Changedetection.io is a self-hosted monitoring service designed to track web pages for content updates and notify users of changes. It functions as a centralized platform where users can manage tracking tasks, observe specific website elements, and receive automated alerts through various communication channels whenever modifications are detected. The service distinguishes itself through an integrated headless browser engine that executes interaction sequences, such as logins or form submissions, to access dynamic or restricted content. It maintains a historical record of page snapshots, util

Photon is a command-line web crawler designed for security reconnaissance and information gathering. It systematically traverses websites to discover URLs, map domain infrastructure, and identify associated subdomains by retrieving DNS records. The tool distinguishes itself through its ability to perform deep content analysis, including the extraction of sensitive data such as API keys and authentication tokens using user-defined regular expressions. It supports offline inspection by cloning crawled web content to the local filesystem, allowing for structural analysis without additional netwo

Pattern is a Python web mining library that functions as an HTML web scraper, a natural language processing toolkit, and a network analysis tool. It provides a mathematical framework for categorizing datasets through a vector space model library. The project enables the extraction of structured data from web services and the creation of searchable web content indexes. It processes unstructured text using sentiment analysis, part-of-speech tagging, and n-gram searching. The library covers machine learning classification through the training of models using perceptron algorithms and support ve

WeChatRobot is a framework providing programmatic interfaces for automating messages and interactions within the WeChat ecosystem. It functions as a bot framework that enables the automation of sending and receiving messages by intercepting system processes. The project includes specialized tools for intercepting system calls and utilizing memory-based application hooking to automate workflows, including support for Enterprise WeChat. It further provides capabilities to decrypt local application databases to recover historical chat logs and user information. Additional functionality covers t

reconftw is an attack surface management framework and reconnaissance workflow orchestrator designed to automate the discovery, mapping, and monitoring of external digital assets. It operates as a modular tool-chain pipeline that coordinates a sequence of security tools to perform intelligence gathering and vulnerability scanning. The project distinguishes itself through a cloud-native deployment model that parallelizes scanning workloads across a fleet of remote VPS instances to bypass local resource constraints. It utilizes container-based environment isolation to ensure consistent executio

Flowsint is an open-source intelligence framework and reconnaissance orchestrator used for cybersecurity investigations. It functions as a containerized tool runner and data mapper, automating the collection of intelligence from open-source providers and APIs to profile targets and map threat intelligence. The platform distinguishes itself through a graph-based investigation interface, where processed raw intelligence is converted into nodes and edges to visualize relationships between entities. It allows for the creation of sequenced pipelines that chain data enrichment tools, enabling the o

This project is a web scraping authentication tool designed to automate login simulations and bypass authentication walls. It provides utilities for programmatically navigating login screens to gain access to protected web content and restricted data. The software specifically enables session establishment through QR code authentication, allowing users to scan a code with a mobile device to capture and save authentication cookies. These captured sessions are used to maintain persistence for future automated requests. The tool covers broader capabilities in browser-based automation, including

Naabu is a port scanner library and tool that probes hosts for open ports using SYN, CONNECT, and UDP methods to identify active services. It functions as a Go library for embedding port scanning into programs, and as a standalone tool that accepts targets as hostnames, IP addresses, CIDR ranges, or ASN numbers. The tool discovers live hosts before scanning, filters ports by range or top lists, and can integrate with Nmap for service version detection. The project distinguishes itself through its SYN-based port probing approach that sends TCP SYN packets and analyzes responses without complet

Waybackurls is a command-line OSINT tool that retrieves every known URL for a given domain from the Wayback Machine archive. It functions as a domain reconnaissance utility, discovering forgotten API endpoints, legacy pages, and hidden files by querying the public web archive API. The tool processes domains independently and statelessly, reading domain names from standard input and streaming discovered URLs line-by-line to standard output. This design enables seamless integration into Unix command pipelines, allowing users to chain waybackurls with other tools for filtering, sorting, and furt

Robin is an AI-powered open source intelligence framework and dark web investigation tool. It functions as a multi-model AI orchestrator that integrates search engines and web scrapers with language models to automate information gathering and data synthesis. The system utilizes a crawl-and-filter architecture to isolate high-value data from raw web content and employs a query-refinement pipeline to optimize search terms. It specifically supports dark web investigations by routing requests through proxies to access hidden services and using language models to analyze and summarize findings fr

Lychee is a command line link checker written in Rust that scans websites and local files to identify broken hyperlinks and invalid email addresses. It functions as an asynchronous link validator and static site auditor designed to ensure content accessibility and quality. The tool features disk-based response caching to accelerate subsequent scans and avoid redundant network requests. It supports token-based API authentication for services like GitHub to bypass rate limits when verifying links on remote repositories. The project covers a range of capabilities including multi-format content

This project is an LLM-powered web crawler and data extractor that uses large language models to navigate websites and parse content into structured JSON or Markdown formats. It functions as an automated browser orchestrator and domain discovery engine, interpreting plain English instructions to identify relevant pages and extract specific information. The system distinguishes itself through agentic browser automation, allowing it to perform human-like interactions such as clicking buttons and scrolling based on natural language commands. It employs goal-oriented crawling to analyze website s

Intelligent web page comparison tool with Wayback Machine support and visual regression testing