hickford/MechanicalSoup
MechanicalSoup
MechanicalSoup est une bibliothèque d'automatisation web Python conçue pour simuler le comportement d'un navigateur. Elle fonctionne comme une boîte à outils pour le web scraping et l'automatisation, fournissant un moteur d'analyse HTML et un gestionnaire de session HTTP pour interagir avec les sites web par programmation.
La bibliothèque permet une interaction web headless en imitant une session utilisateur réelle. Elle gère l'état persistant via la gestion des cookies et le suivi automatique des redirections, permettant la navigation programmatique sur les sites web et la simulation d'interactions complexes avec le navigateur.
Ses capacités couvrent le remplissage et la soumission automatisés de formulaires en utilisant des sélecteurs CSS, ainsi que l'extraction de données à partir de réponses HTML. L'ensemble d'outils inclut des utilitaires pour télécharger des fichiers liés, spécifier des agents utilisateur personnalisés et rechercher des pages basées sur des mots-clés spécifiques. Elle fournit également des outils de diagnostic pour rendre l'état actuel de la page dans un navigateur pour une vérification visuelle.
Features
- Web Scraping and Automation - Provides a comprehensive system for automating browser interactions and crawling web content for data extraction.
- HTML Parsing - Includes an engine for extracting data from web pages by parsing HTML responses.
- DOM Tree Construction - Converts raw HTML responses into a searchable DOM tree structure for targeted data extraction.
- Hyperlink Navigators - Locates specific links via regular expressions or tags and navigates directly to their destinations.
- HTML Parsing and Extraction - Parses HTML responses into searchable tree structures to locate and extract specific tags or text elements.
- Python Browser Automation Libraries - A Python toolkit designed for simulating browser behavior, managing cookies, and automating form submissions.
- Browser-Mimicking Request Simulators - Mimics a real user session by managing cookies, redirects, and user agents across requests.
- Stateful Session Persistence - Maintains browser context and authentication state across multi-step web interactions by tracking cookies and URLs.
- HTTP Cookie Persistence - Maintains session state across requests by storing and retrieving HTTP cookies in a local store.
- Element Targeting - Provides the ability to locate and interact with specific HTML forms and page elements using standardized CSS selectors.
- Data Extractions - Uses CSS selectors and DOM traversal to retrieve specific text, links, or attributes from HTML documents.
- Automated Form Fillers - Fills out and submits web forms programmatically to interact with online services.
- Headless Browsers - Provides a headless interface for interacting with web pages and retrieving content without a graphical browser.
- Session and Cookie Automators - Manages authenticated sessions through the automated handling of HTTP cookies and redirects.
- Programmatic Navigation - Allows moving through a website programmatically by following hyperlinks and searching for keywords.
- Session State Management - Stores cookies and tracks URLs across multiple requests to maintain a persistent user session.
- Web Scraping - Provides utilities for navigating websites, following hyperlinks, and extracting structured data from HTML.
- Automatic Redirect Handlers - Automatically follows HTTP redirection chains to reach the final destination page during navigation.
- Custom User Agent Setters - Allows specifying custom identity strings for HTTP requests to control how servers identify the client.
- Programmatic Navigation - Opens URLs and follows links using text patterns or regular expressions to move through a site.
- Form Data Population - Provides mechanisms for programmatically assigning values to multiple web form fields from data sources.
- Form Data Submission - Selects HTML forms via CSS selectors and populates input fields before triggering a submission.
- HTTP Request and Response Handlers - Handles the lifecycle of sending HTTP requests with custom headers and processing the resulting page content.
- Pattern-Based Link Resolution - Identifies and follows hyperlinks by matching page text against regular expressions or specific tag attributes.
- Python Crawling Frameworks - Automates website interactions for scraping.
- Simulated Browser Environments - Python library for automating website interactions.
- Web Scraping - Automated interaction with websites.
