psf/requests-html

Features

• Web Scraping - Provides a framework for extracting structured information from websites that require JavaScript execution.
• Web Page Retrievers - Fetches HTML content from URLs and automatically handles character encoding for correct text decoding.
• HTML Parsing - Provides a human-friendly Python interface for fetching and parsing HTML web pages.
• Web Page Scraping Extractors - Extracts and structures data from web pages by parsing raw HTML via a human-friendly interface.
• JavaScript Rendering - Executes JavaScript on a web page using a headless browser to retrieve content generated after initial load.
• Structured Data Extraction - Uses CSS selectors and XPath to programmatically isolate and pull specific text or attributes from HTML structures.
• Asynchronous HTTP Clients - Functions as an asynchronous HTTP client that fetches multiple web pages concurrently to improve collection speed.
• CSS Selector Engines - Provides a CSS selector engine to map high-level queries to DOM elements for data extraction.
• Headless Browsers - Integrates a headless browser to execute JavaScript and capture the final rendered state of web pages.
• Headless Rendering Engines - Integrates a headless rendering engine to execute JavaScript and retrieve dynamically generated content.
• Asynchronous Fetching - Provides asynchronous support to fetch multiple web pages concurrently, significantly reducing total network wait time.
• Web Scraping Frameworks - Offers a complete toolset for automating data collection from websites using CSS selectors and XPath.
• Web Scraping Selectors - Locates specific HTML elements using CSS selectors or XPath expressions to retrieve text and attributes.
• Hyperlink Extraction - Collects all hyperlinks from a page and provides them as relative or absolute URLs.
• Connection Pooling - Maintains a set of reusable TCP connections to minimize the overhead of opening sockets to the same host.
• Session Management - Maintains cookies and implements connection pooling to preserve state and handle redirects across multiple requests.
• Event Loop Concurrency - Implements an asynchronous event loop to handle multiple network requests simultaneously without blocking the main thread.
• Stateful Session Persistence - Persists cookies and headers across multiple requests to simulate a continuous user session.
• Web Session Management - Handles cookies and connection pooling to maintain a consistent state across multiple requests to a server.
• Hyperlink Harvesting - Automatically collects and normalizes all hyperlinks from a website to map page structures or build crawlers.

Star history