Open-source alternatives to Robobrowser

MechanicalSoup is a Python web automation library and scraping framework designed to simulate browser sessions and navigate websites without requiring JavaScript execution. It functions as an HTML parsing tool and HTTP session manager, allowing for the programmatic retrieval of page content and the automation of web interactions. The library distinguishes itself by combining session persistence with automated form interaction. It maps user data to HTML input fields and selection boxes for programmatic submission and maintains authenticated states by managing cookies and user-agent headers acr

This project is a Python web scraping library and automated data collection suite. It provides tools for extracting structured data from websites, implementing web crawlers to navigate site links, and parsing HTML DOM structures to isolate specific elements and attributes. The toolkit includes a pipeline for processing unstructured text and cleaning raw web content to extract meaningful information. It also features capabilities for image data extraction and the integration of external APIs to retrieve structured data from remote endpoints. The system covers broad capability areas including

AngleSharp is an HTML5 DOM parser and web scraping framework designed to parse HTML5, SVG, and MathML documents into a W3C compliant document object model. It functions as a programmatic HTML generator and a CSS selector engine for querying and locating specific elements within a DOM. The project provides tools for simulating browser environments to automate web interactions, navigate URLs, and submit forms. It includes a dedicated HTML and CSS minifier to reduce the file size of web assets by removing unnecessary characters. The library supports HTML DOM manipulation and the extraction of s

scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields. The tool uses CSS selectors to target specific data points and employs schema-driven data mapping to organize unstructured web text into a consistent format. It supports custom value transformation to convert raw extracted strings into specific data formats. The system provides capabilities for web data extraction and automated cont

parse5 is a WHATWG HTML parser and serializer for Node.js. It transforms HTML strings into a document object model and converts those trees back into valid HTML strings, following the logic defined by the HTML Living Standard. The project functions as a streaming HTML processor, using incremental parsing to handle large documents in chunks. It includes an HTML5 compliant tokenizer that uses a state-machine approach to break input into tokens according to official web specifications. The toolset covers HTML document parsing, serialization, and real-time rewriting via streams. These capabiliti

PHP HTML Parser is a server-side programming library and DOM parser designed to ingest markup documents and structure them into a navigable parent-child object tree. The library provides a stream-based character lexer and a configurable rule engine that manages parser strictness, whitespace preservation, tag closing behavior, and character encoding detection during document ingestion. Content loading is handled through a pluggable retrieval layer that accepts local file paths, raw strings, and remote network URLs. Once loaded, documents can be queried using a chainable CSS selector engine th

SwiftSoup is a cross-platform HTML processing library for Swift that converts raw HTML or XML strings and files into a structured document object model. It provides the core infrastructure to parse web content into a traversable tree, enabling programmatic access to page elements across iOS, macOS, and Linux. The library features a CSS selector engine for data extraction and a whitelist-based sanitization system to remove unsafe tags and attributes from user-submitted content. It optimizes repetitive document queries through memoized query caching. The project covers DOM manipulation for upd

jsdom is a Node.js implementation of web standards that functions as a headless browser emulator. It provides a JavaScript execution environment and an HTML and XML parser to simulate a browser environment on the server side, implementing various web APIs and W3C standards. The project distinguishes itself by providing a sandboxed runtime for executing scripts embedded in HTML or external files. It includes specialized polyfills for the Canvas API and manages session state through HTTP cookie management. Its broader capabilities cover network interaction via request interception and resource

This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data. The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic

Playwright for Python is a browser automation framework designed for end-to-end testing, web scraping, and user interaction simulation. It functions as a headless browser controller that enables programmatic navigation, data extraction, and the execution of complex workflows across multiple rendering engines. The framework distinguishes itself through an actionability-aware interaction engine that automatically verifies element readiness before performing actions, significantly reducing test flakiness. It utilizes isolated browser contexts to maintain separate storage and cookies for parallel

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

This project is a command-line utility designed to fetch video, audio, and image content from a wide range of web platforms. It functions by parsing page metadata and utilizing modular, site-specific scripts to extract direct media stream URLs from complex web structures, enabling the local archiving of digital media for offline use. The tool distinguishes itself through its ability to handle authenticated content, allowing users to inject browser-stored session cookies to access restricted or private media. It also supports real-time media streaming by piping remote content directly into ext

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

PyQuery-based scraping micro-framework.

A high-level distributed crawling framework.

Scraperr is a self-hosted web scraping and crawling platform designed for extracting structured data from websites using XPath selectors. It functions as a containerized system for managing scraping jobs through a queue and analyzing the resulting content using artificial intelligence. The project differentiates itself through its Kubernetes-native architecture, allowing for scalable deployment and management via package managers. It includes a crawling engine capable of domain-level spidering to discover linked pages and a data analyzer that uses artificial intelligence to query extracted we

goquery is a Go HTML parsing library and CSS selector engine used to isolate and retrieve specific text or attributes from HTML documents. It functions as an HTML DOM manipulator that converts raw HTML strings into a structured tree for programmatic navigation and search. The library provides a fluent interface for chaining selection and filtering operations and utilizes a wrapper-based abstraction to simplify data extraction and manipulation of nodes. It employs an iterator-based processing mechanism to apply operations to every node within a matched selection. Its primary capabilities cove

HTML-Renderer is a managed C# library that processes HTML and CSS content to render desktop user interfaces, generate image files, and export PDF documents. Built entirely in managed code without external native dependencies, the library parses markup into a structured document object model, applies cascading style rules, and computes virtual box layouts directly in memory. The rendering pipeline features a direct bitmap rasterisation engine that draws styled layouts straight onto graphics targets and document canvases, along with a dedicated pagination engine that splits continuous layouts a

HTMLHint is an HTML linter and static analysis tool designed to identify structural errors and syntax violations in source code. It functions as a configurable validator that scans local files and remote URLs to detect malformed tags, duplicate IDs, and document structure inconsistencies. The tool serves as an accessibility auditor, verifying that web elements include necessary labels and alternative text for images. It further distinguishes itself by enforcing web document compliance, checking for the presence of doctypes, language attributes, and head metadata. The project covers a broad r

This project is a collection of Python scripts and source code examples designed for learning programming fundamentals through practical application. It serves as a toolkit for web scraping and browser automation, alongside a library of utilities for data processing. The repository includes scripts for simulating human interactions to automate repetitive web tasks and online booking processes. It also provides a structured database of administrative divisions, including provinces, cities, and districts, for geographic data management and address validation. The collection covers capabilities

python-goose is a Python library for web scraping and content extraction. It functions as an HTML boilerplate remover and article parser designed to isolate primary text and metadata from web pages by stripping away navigation, layout noise, and non-essential elements. The tool features multilingual processing capabilities, utilizing language-specific stop-word analyzers to identify and extract primary content across different languages. It also identifies and collects embedded media, including source URLs and embed codes for lead images and videos associated with an article. The library cov

Panel is a Python data application framework used to build interactive dashboards and reactive user interfaces by connecting data visualizations, widgets, and callbacks. It functions as a reactive UI library that synchronizes application state with interface updates based on user interactions. The project distinguishes itself by offering multiple execution models, including a FastAPI-based web application server for scalable session management and a WebAssembly compiler that enables Python applications to run directly in the browser without a backend server. It also includes a specialized too

CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage. The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network

php-webdriver is a WebDriver PHP client and browser automation framework that implements the W3C WebDriver standard. It serves as a programmatic interface for controlling web browsers, executing JavaScript, and managing browser sessions in both headed and headless environments. The library functions as a Selenium protocol implementation, allowing PHP applications to communicate with browser drivers such as ChromeDriver or GeckoDriver. It provides the ability to automate user actions, navigate pages, and validate DOM elements for web UI testing. Its capabilities cover broad areas of browser i