psf/requests-html
Requests Html
requests-html is a Python HTML parsing library and web scraping framework. It functions as an asynchronous HTTP client and a JavaScript rendering engine designed to fetch and parse web pages for structured data extraction.
The project integrates a headless browser to execute JavaScript, allowing it to retrieve dynamically generated content that standard HTML parsers cannot see. It provides tools for automated data extraction using CSS selectors and XPath expressions to isolate specific text or attributes from HTML structures.
The framework covers network operations including asynchronous page fetching, session state management with cookies, and connection pooling. It also includes utilities for hyperlink retrieval to harvest and normalize URLs from websites.
Features
- Web Scraping - Provides a framework for extracting structured information from websites that require JavaScript execution.
- Web Page Retrievers - Fetches HTML content from URLs and automatically handles character encoding for correct text decoding.
- HTML Parsing - Provides a human-friendly Python interface for fetching and parsing HTML web pages.
- Web Page Scraping Extractors - Extracts and structures data from web pages by parsing raw HTML via a human-friendly interface.
- JavaScript Rendering - Executes JavaScript on a web page using a headless browser to retrieve content generated after initial load.
- Structured Data Extraction - Uses CSS selectors and XPath to programmatically isolate and pull specific text or attributes from HTML structures.
- Asynchronous HTTP Clients - Functions as an asynchronous HTTP client that fetches multiple web pages concurrently to improve collection speed.
- CSS Selector Engines - Provides a CSS selector engine to map high-level queries to DOM elements for data extraction.
- Headless Browsers - Integrates a headless browser to execute JavaScript and capture the final rendered state of web pages.
- Headless Rendering Engines - Integrates a headless rendering engine to execute JavaScript and retrieve dynamically generated content.
- Asynchronous Fetching - Provides asynchronous support to fetch multiple web pages concurrently, significantly reducing total network wait time.
- Web Scraping Frameworks - Offers a complete toolset for automating data collection from websites using CSS selectors and XPath.
- Web Scraping Selectors - Locates specific HTML elements using CSS selectors or XPath expressions to retrieve text and attributes.
- Hyperlink Extraction - Collects all hyperlinks from a page and provides them as relative or absolute URLs.
- Connection Pooling - Maintains a set of reusable TCP connections to minimize the overhead of opening sockets to the same host.
- Session Management - Maintains cookies and implements connection pooling to preserve state and handle redirects across multiple requests.
- Event Loop Concurrency - Implements an asynchronous event loop to handle multiple network requests simultaneously without blocking the main thread.
- Stateful Session Persistence - Persists cookies and headers across multiple requests to simulate a continuous user session.
- Web Session Management - Handles cookies and connection pooling to maintain a consistent state across multiple requests to a server.
- Hyperlink Harvesting - Automatically collects and normalizes all hyperlinks from a website to map page structures or build crawlers.
