Open-source alternatives to CeWL

Crawler4j is a multi-threaded Java web crawler and spider designed for high-volume web traversal and content extraction. It functions as a polite crawling framework that enables the discovery and indexing of HTML and binary content across multiple websites. The project distinguishes itself through a persistent crawling model that serializes session state to local storage, allowing the engine to resume indexing after a crash or interruption. It includes a politeness controller to regulate request frequency and delays, preventing server overloading and IP blocking. The system covers a broad ra

X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It functions as an HTML data extractor that transforms raw page content into a defined schema using CSS-style selectors. The project implements a headless browser crawler capable of executing JavaScript to render dynamic content. It handles website content discovery through a breadth-first crawling strategy and automatic pagination discovery to traverse multi-page result sets. The framework manages web data pipelines using a concurrency-limited request queue and request rate cont

This repository is a collection of Java backend projects, encompassing a web application suite, a real-time chat server, a web crawler, and an authentication system. It provides implementations for building server-side logic and business rules to process data and handle web requests. The project includes a networked messaging system for instant bidirectional communication and an automated web scraping engine that parses HTML to aggregate data into local storage. It also features a security framework for managing user permissions and generating visual captchas to prevent automated bot access.

fsociety is a penetration testing framework and security tool orchestrator designed to conduct full security audits. It functions as a wrapper that integrates external security binaries into a unified, menu-driven interface, providing a centralized system for command-line parameter mapping and execution. The project distinguishes itself by organizing specialized utilities into domain-specific collections for structured navigation. It automates the transition between different phases of an audit by chaining reconnaissance and exploitation tools through sequential workflow automation. The fram

Jodd is a suite of lightweight Java extensions and standard library utilities designed for application configuration, database mapping, dependency injection, and HTML parsing. It provides a consolidated set of core tools to facilitate Java development with a zero-dependency core to ensure compatibility and a small footprint across environments. The project features a pragmatic dependency injection container for managing object lifecycles and a database mapper that uses SQL templates to map result sets directly to Java objects. It includes a specialized configuration manager supporting profile

JMComic-Crawler-Python is a high-performance asynchronous web scraper and API client designed to programmatically retrieve images and metadata from a comic hosting service. It functions as a media archiving tool for batch downloading albums and chapters, automating the process of saving content to a local filesystem. The project is distinguished by its ability to reverse server-side pixel obfuscation, using a decryption tool to reconstruct sliced and shuffled images. To maintain stable connectivity, it utilizes a network bypass utility featuring dynamic domain rotation and proxy routing to ci

This project is a comprehensive web application penetration testing guide and vulnerability research framework. It provides a structured methodology for identifying and exploiting security flaws through a phased approach involving reconnaissance, analysis, and exploitation. The resource is distinguished by its use of a curated methodology framework that links theoretical vulnerability patterns to real-world bug bounty reports and historical exploit examples. It includes a payload-based testing library and a reference system that maps specific vulnerability categories to recommended third-part

MechanicalSoup is a Python web automation library and scraping framework designed to simulate browser sessions and navigate websites without requiring JavaScript execution. It functions as an HTML parsing tool and HTTP session manager, allowing for the programmatic retrieval of page content and the automation of web interactions. The library distinguishes itself by combining session persistence with automated form interaction. It maps user data to HTML input fields and selection boxes for programmatic submission and maintains authenticated states by managing cookies and user-agent headers acr

Knwl.js is a JavaScript named entity recognition library and text entity extractor. It functions as an extensible text parsing engine designed to scan unstructured strings for specific data patterns and convert them into structured information. The engine utilizes a modular framework that allows for the recognition of custom data types. This is achieved through a plugin system for language pattern matching, enabling the integration of custom logic to identify unique data types within text. The library identifies and isolates entities such as dates, times, phone numbers, emails, and locations

ShuiZe_0x727 is an open-source intelligence gathering framework and attack surface management tool. It functions as an asset discovery engine and cyber intelligence aggregator designed to identify internet-facing assets, map network infrastructure, and visualize total network exposure. The project integrates vulnerability scanning and sensitive data leak detection to identify security weaknesses and unauthorized access points. It employs a combination of network space API queries, certificate log analysis, and public repository scanning to extract leaked credentials, API keys, and internal ad

MechanicalSoup is a Python web automation library designed to simulate browser behavior. It functions as a toolkit for web scraping and automation, providing an HTML parsing engine and an HTTP session manager to interact with websites programmatically. The library enables headless web interaction by mimicking a real user session. It manages persistent state through cookie handling and automatic redirect following, allowing for programmatic website navigation and the simulation of complex browser interactions. Its capabilities cover automated form population and submission using CSS selectors

Learn-Web-Hacking is a structured web security study guide and penetration testing knowledge base. It provides a collection of research notes focused on identifying and exploiting vulnerabilities in web applications and network protocols. The project includes specialized frameworks for evaluating security risks in large language models to prevent prompt injection, as well as guides for hardening cloud-native infrastructure, including container standards and orchestration tools. It also covers the analysis of identity standards and authentication protocols. The material spans a broad range of

Flan is a containerized network vulnerability scanner and security auditor. It identifies open ports and service versions across a network to detect known security weaknesses and misconfigurations. The system is designed to run within isolated container environments, utilizing configuration maps to manage target lists and secrets. It includes a dedicated mechanism for archiving scan output files and security analysis data to remote S3 buckets for long-term storage. The tool generates formatted vulnerability summaries and security reports in multiple document formats for technical analysis. I

This project is a comprehensive API security audit checklist and vulnerability audit framework. It provides a structured guide of security countermeasures for designing, testing, and deploying secure APIs across various protocols. The framework includes specialized guides for securing OAuth 2.0 authorization flows, implementing zero trust networking for service-to-service communication, and protecting GraphQL endpoints from resource exhaustion and information leakage. It also provides standards for integrating static analysis, dynamic scanning, and secret detection into CI/CD delivery pipelin

AllHackingTools is a security tool orchestrator and suite designed to install, update, and manage a wide array of third-party hacking and security utilities from a single command interface. It functions as a centralized hub for network analysis, open source intelligence, penetration testing, and social engineering tools. The project provides specialized frameworks for gathering open source intelligence and searching for user profiles across social platforms. It includes toolkits for network reconnaissance, vulnerability scanning, and the execution of security exploits, as well as a social eng

XSStrike is an automated security scanning engine designed for web application discovery, input

Lynis is an automated security auditing and system hardening framework designed for UNIX-based operating systems. It functions as a command-line utility that inspects local system configurations to identify security vulnerabilities, configuration weaknesses, and compliance gaps. By executing a series of modular tests, the tool generates actionable reports and remediation suggestions to assist in strengthening system defenses. The project distinguishes itself through a highly modular architecture that relies on shell-script-based execution and native system inspection. Users can define custom

This project provides a comprehensive, modular framework for auditing and hardening personal digital and physical security. It functions as a structured, platform-agnostic knowledge base that breaks down complex security standards into granular, actionable tasks. By utilizing a static documentation architecture, the project ensures that its guidance remains accessible and transparent, allowing users to track their security posture incrementally through a persistent, manual progress-tracking system. The project distinguishes itself by bridging the gap between digital cybersecurity and physical

Lux is a command line video downloader written in Go designed for extracting and saving video and audio from various websites. It functions as a concurrent media downloader that increases transfer speeds by splitting files into fragments and downloading them using multiple threads. The tool serves as a playlist download manager capable of retrieving entire video collections or specific ranges of items. It also operates as a proxy-enabled media client, supporting HTTP and SOCKS5 proxies and session cookies to access region-locked, private, or age-gated content. Additional capabilities include

Go Spider is a modular framework designed for building concurrent web scrapers and data extraction workflows. It provides a structured engine for orchestrating automated crawling tasks, managing request scheduling, and processing web content through a unified pipeline. The framework distinguishes itself through a highly configurable architecture that allows developers to inject custom logic for downloaders, schedulers, and storage components via interface-driven contracts. It manages network interactions using middleware-based request throttling and URL deduplication, ensuring that crawling o

hacker-scripts is a multi-purpose toolkit comprising a security vulnerability testing suite, a keyword-driven email automator, and a TCP remote hardware controller. It provides a collection of scripts for identifying security weaknesses and conducting controlled hacking experiments across multiple programming language environments. The system automates email workflows by scanning headers and bodies for specific strings to trigger server responses and sends randomized text and email alerts based on system activity. It also enables remote hardware control by sending binary commands over network

UACME is a set of specialized tools designed to audit security configurations, escalate user privileges, and circumvent access control restrictions on Windows systems. It functions as a utility for executing commands with elevated privileges by bypassing User Account Control restrictions. The project includes a configuration auditor used to extract and analyze system settings to identify security misconfigurations and vulnerabilities. It provides a collection of techniques for gaining administrative rights on a host. The toolset covers a wide range of privilege escalation and security auditi

d3-cloud is a JavaScript library for creating non-overlapping word cloud layouts using the D3.js data visualization framework. It serves as a text visualization tool that converts raw text or CSV data into scalable vector graphics and stylized typographic clouds. The library utilizes a spiral-based collision layout algorithm to compute coordinates and rotations, ensuring that words within a visual cluster do not overlap. It maps word frequency to font size and supports a customized layout process where positions are calculated based on text distribution. The system covers data processing for

This project is a reference library of architectural blueprints, study materials, and design patterns for building scalable, high-availability distributed systems. It serves as a technical guide for scalability engineering, providing structural solutions for common engineering challenges. The repository focuses on distributed systems design, covering essential patterns for data replication, consensus algorithms, and transaction management. It distinguishes itself by offering detailed blueprints for specialized domains, including real-time data streaming, large-scale data storage, and high-ava

Bustag is a containerized media library manager and automated metadata aggregator. It serves as a centralized database for tracking digital assets, utilizing machine learning models to act as a predictive content filter that identifies and recommends media based on user preferences. The system distinguishes itself through a supervised preference modeling workflow, where users manually label assets as liked or disliked to train predictive models. These models then automate content curation and personalized recommendations by analyzing labeled datasets. The platform integrates web data aggrega

CDK is a specialized toolset for container security auditing, container escape exploitation, and cloud infrastructure pentesting. It provides a collection of scripts and tools designed to identify and exploit vulnerabilities in container runtimes to break out of isolated environments and execute commands on the underlying host operating system. The project features a dedicated Docker runtime exploit suite for abusing the Docker API, procfs, and cgroups to gain unauthorized host-level access. It includes specific techniques for bypassing isolation via LXCFS, user namespace exploitation, and ho

This project is a web application security guide and developer training resource. It serves as a secure coding framework and vulnerability remediation manual, providing software engineers with the tools to identify, prioritize, and fix common security holes across different application layers. The resource utilizes a structured verification framework and security audit checklists to systematically find vulnerabilities. It features a technical reference that maps specific security flaws to step-by-step instructions for remediation, supported by vulnerability statistics to help determine which