Requests Html

Features

Web Scraping - Provides a framework for extracting structured information from websites that require JavaScript execution.
Web Page Retrievers - Fetches HTML content from URLs and automatically handles character encoding for correct text decoding.
HTML Parsing - Provides a human-friendly Python interface for fetching and parsing HTML web pages.
Web Page Scraping Extractors - Extracts and structures data from web pages by parsing raw HTML via a human-friendly interface.
JavaScript Rendering - Executes JavaScript on a web page using a headless browser to retrieve content generated after initial load.
Structured Data Extraction - Uses CSS selectors and XPath to programmatically isolate and pull specific text or attributes from HTML structures.
Asynchronous HTTP Clients - Functions as an asynchronous HTTP client that fetches multiple web pages concurrently to improve collection speed.
CSS Selector Engines - Provides a CSS selector engine to map high-level queries to DOM elements for data extraction.
Headless Browsers - Integrates a headless browser to execute JavaScript and capture the final rendered state of web pages.
Headless Rendering Engines - Integrates a headless rendering engine to execute JavaScript and retrieve dynamically generated content.
Asynchronous Fetching - Provides asynchronous support to fetch multiple web pages concurrently, significantly reducing total network wait time.
Web Scraping Frameworks - Offers a complete toolset for automating data collection from websites using CSS selectors and XPath.
Web Scraping Selectors - Locates specific HTML elements using CSS selectors or XPath expressions to retrieve text and attributes.
Hyperlink Extraction - Collects all hyperlinks from a page and provides them as relative or absolute URLs.
Connection Pooling - Maintains a set of reusable TCP connections to minimize the overhead of opening sockets to the same host.
Session Management - Maintains cookies and implements connection pooling to preserve state and handle redirects across multiple requests.
Event Loop Concurrency - Implements an asynchronous event loop to handle multiple network requests simultaneously without blocking the main thread.
Stateful Session Persistence - Persists cookies and headers across multiple requests to simulate a continuous user session.
Web Session Management - Handles cookies and connection pooling to maintain a consistent state across multiple requests to a server.
Hyperlink Harvesting - Automatically collects and normalizes all hyperlinks from a website to map page structures or build crawlers.

Open-source alternatives to Requests Html

Similar open-source projects, ranked by how many features they share with Requests Html.

bjesus/pipet
bjesus/pipet
4,662View on GitHub
pipet is a command-line tool that turns web scraping into a piped data flow through Unix filters. It provides a set of specialized scrapers — for CSS selector extraction, headless browser JavaScript rendering, JSON API querying, and change monitoring — each outputting structured data that can be transformed by chaining additional commands. The tool uses declarative selectors (CSS and JSON path expressions) to define what to extract, automatically follows pagination links to collect data across multiple pages, and serializes results into JSON, custom-delimited text, or rendered templates. It c
Gocsscurlgjson
View on GitHub4,662
apify/crawlee-python
apify/crawlee-python
8,097View on GitHub
Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra
Pythonapifyautomationbeautifulsoup
View on GitHub8,097
apify/crawlee
apify/crawlee
24,002View on GitHub
Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
TypeScriptapifyautomationcrawler
View on GitHub24,002
wistbean/learn_python3_spider
wistbean/learn_python3_spider
21,802View on GitHub
This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac
Pythonpython-scriptpython-spiderpython3
View on GitHub21,802

See all 30 alternatives to Requests Html

psfrequests-html

Features

Open-source alternatives to Requests Html

bjesus/pipet

apify/crawlee-python

apify/crawlee

wistbean/learn_python3_spider

Star history

Open-source alternatives to Requests Html

bjesus/pipet

apify/crawlee-python

apify/crawlee

wistbean/learn_python3_spider