autoscrape-labs/pydoll

Features

• Browser Protocol Controllers - Controls Chromium browsers via direct low-level debugger protocol commands over WebSockets.
• Browser Automation Protocols - Controls Chromium-based browsers via direct Chrome DevTools Protocol commands to bypass WebDriver overhead and detection.
• Headless Browsers - Implements a headless browser controller for executing scripts and rendering pages without a visible user interface.
• DevTools Protocol Clients - Acts as a client that communicates with Chromium browsers via the Chrome DevTools Protocol for automation and control.
• DOM Element State Retrieval - Allows fetching current text and bounding box coordinates directly from the browser to reflect real-time page updates.
• Keyboard and Mouse Input Simulations - Simulates a full range of keyboard, mouse, touch, and gesture events to mimic human interaction.
• Human-Like Interaction Simulators - Simulates human-like mouse movements and timing using Fitts's Law parameters to evade bot detection.
• DOM-to-Model Mappings - Maps web page content to typed models using selectors, attributes, and custom transformations.
• Browser Session Data Management - Enables programmatic management of browser-specific session data including cookies and local storage for persistence.
• Metadata Extraction - Retrieves detailed metadata including attributes, box model dimensions, and computed styles for DOM elements.
• Web Data Extraction Tools - Collects structured data from dynamic web pages, including those utilizing shadow DOM or nested iframes.
• Browser-Context Script Executions - Executes scripts in a global context or bound to a specific element with environment compatibility.
• CDP Target Attachments - Provides controls for attaching to and detaching from specific browser targets and configuring auto-attachment.
• Protocol Command Execution - Provides low-level control over the browser through direct debugger protocol command execution.
• Event-Driven Automation Engines - Provides a built-in event system to react in real-time to page, network, and runtime changes.
• Human-Mimetic Input Simulations - Mimics real human typing by introducing variable delays, cognitive pauses, and realistic typos.
• Browser Target Discovery - Enables retrieval of active targets and monitoring of the creation and destruction of tabs and workers.
• Human-Like Path Generation - Mimics natural human cursor movement using Bezier curves and easing to evade bot detection.
• Bidirectional Socket Communication - Handles real-time asynchronous messaging between the host and browser using WebSockets for event-driven callbacks.
• Iframe Focus Switching - Automatically resolves iframe contexts to manipulate elements without manual context switching.
• Network Request Interceptors - Intercepts and modifies network traffic in real time to mock server responses and alter HTTP requests.
• Programmable Request Interceptors - Provides the ability to programmatically pause and modify outgoing requests and incoming responses.
• Anti-Bot Evasion - Implements a toolkit that mimics human mouse movements and typing patterns to evade behavioral bot detection.
• Spoofing Tools - Overrides JavaScript properties and canvas rendering via protocol commands to mask automation and prevent fingerprinting.
• Browser Identity Synchronization - Aligns HTTP User-Agent headers with browser navigator properties and Client Hints to ensure a consistent identity.
• Automated Captcha Solvers - Automates the solving of Cloudflare Turnstile and reCAPTCHA v3 challenges without external APIs.
• Captcha Interaction Simulations - Simulates browser clicks on checkbox captchas to trigger verification based on environment trust scores.
• Cookie Management - Provides granular control over creating, retrieving, and updating browser cookies including domain and security policies.
• Fingerprint Configuration - Configures internal browser preferences and settings to maintain consistent and realistic device identities.
• Same-Origin Policy Relaxation - Bypasses same-origin policies to enable interaction and data extraction from cross-domain iframes.
• Human Behavior Simulations - Simulates natural interaction patterns and keystroke timing to evade behavioral analysis and bot detection.
• Mouse Movement Simulations - Generates natural cursor movements using Bezier curves and physiological tremors to mimic human motion.
• Protocol Message Batching - Groups multiple protocol instructions into single messages to minimize network round trips and increase speed.
• Network Response Modifiers - Provides a network proxy to intercept and modify incoming responses and outgoing requests in real time.
• Request Interception Utilities - Modifies the URL, method, headers, or body of paused network requests in real time.
• Element Interaction - Implements high-level APIs for simulating user input on web elements using stable browser identifiers.
• Browser Context Isolation - Creates separate browsing environments with independent cookies and storage for session isolation.
• Iframe Navigators - Provides utilities to locate and interact with elements nested within iframe chains.
• DOM Element Selectors - Enables finding page elements using CSS selectors, XPath, plain text, and spatial coordinates.
• Element Locators - Identifies nested elements by searching within the scope of a previously located element.
• Element Click Simulators - Simulates mouse clicks on page elements using viewport coordinates and button specifications for automation.
• Humanized Scrolling - Mimics natural browsing behavior by scrolling with physics-based momentum and jitter.
• Browser Storage - Reads and writes cookies and local storage data to maintain persistent session states.
• Isolated Browser Contexts - Creates independent storage and session partitions within a single browser process to isolate contexts.
• Tab Management - Provides comprehensive tools for monitoring and manipulating browser tab states and lifecycles.
• Browser Lifecycle Management - Manages page navigation and lifecycle states to ensure the browser is ready for automation.
• Browser Navigations - Provides the ability to direct the browser to specific URLs with custom referrers and transition configurations.
• Concurrent Browser Contexts - Manages multiple isolated browser contexts and tabs concurrently for parallel automation tasks.
• Browser Session Persistence - Implements mechanisms to maintain and reuse browser sessions for automation and bypassing security challenges.
• Data Extractions - Retrieves text content, bounding box coordinates, and DOM references for web data extraction.
• Headless Browser Controllers - Provides a control layer for managing isolated headless browser contexts and tabs via direct WebSocket connections.
• Shadow DOM Traversal - Navigates shadow roots attached to elements with support for polling to interact with encapsulated components.
• Session Profile Isolations - Manages distinct browser user profiles using temporary directories to ensure complete session isolation.
• Shadow DOM Interactors - Enables querying and manipulation of elements within shadow roots and cross-origin iframes, including closed roots.
• Browser Lifecycle Managers - Controls the startup, shutdown, and protocol connection states of the browser process.
• Remote Browser Connections - Connects to existing browser instances via WebSockets for use in containers or automation pipelines.
• Protocol Connection Management - Establishes and maintains WebSocket connections to browser-level or page-level endpoints.
• Keyboard Shortcut Simulations - Simulates the combination of multiple keys into single actions for common tasks like copying and pasting.
• Type Conversions - Converts raw DOM strings into specific types like numbers or dates using custom transformation callables.
• Hierarchical Extraction - Resolves hierarchical data by defining scope elements to extract nested sub-models from the DOM.
• Browser Download Automations - Captures triggered browser downloads and saves them to specified or temporary directories.
• Browser Download Management - Supports programmatic triggering and tracking of file downloads to specific local paths.
• Concurrent Task Execution - Executes multiple automation tasks across different browser tabs simultaneously to increase throughput.
• Runtime Request Modification - Allows real-time modification of request headers, URLs, and bodies during active browser sessions.
• Localization Configurations - Provides utilities for managing default language settings and regional simulation in the browser.
• Network Connectivity Toggles - Simulates network conditions and toggles browser caching to test application resilience and connectivity.
• Response Body Parsing - Retrieves response bodies as structured data including text, bytes, and parsed JSON.
• DOM Traversers - Provides algorithms for navigating DOM hierarchies to retrieve child and sibling elements.
• Element-Specific Captures - Allows capturing images of specific UI elements by automatically scrolling them into the viewport.
• Traffic Captures - Records all network activity on a tab and exports it as a HAR file with resource type filtering.
• HTTP Header Spoofing - Intercepts and forces consistency across HTTP headers to prevent browser identification.
• Context-Specific Proxy Assignments - Allows the configuration of unique proxy servers and credentials for individual browser contexts.
• Response Body Processing - Provides mechanisms to retrieve the actual content of completed network responses using request identifiers.
• API Response Simulators - Provides tools to intercept and stub API responses to simulate various backend server scenarios.
• Proxy Configurations - Manages proxy URLs and authentication credentials to route browser traffic through specific gateways.
• Request Header Overrides - Applies custom HTTP request headers to all outgoing traffic to test server-side logic and bypass restrictions.
• DOM Tree Traversers - Implements recursive traversal of the document tree to locate parent, child, and detached nodes.
• Bypasses - Toggles the ability to bypass page security policies to execute restricted scripts.
• Device Model Simulation - Simulates different device hardware models by modifying user-agent strings and platform metadata.
• Script Execution Isolators - Establishes isolated execution environments for scripts within specific frames to prevent interference.
• Tracking Prevention - Adjusts anti-tracking settings including Do Not Track and third-party cookie blocking.
• SSL Certificate Validation Overrides - Disables strict TLS/SSL certificate checking to allow connections to invalid certificates.
• VPN Traffic Leak Prevention - Blocks WebRTC and DNS prefetch leaks to ensure all traffic is routed exclusively through a proxy.
• WebRTC IP Leak Protections - Prevents the exposure of local IP addresses via WebRTC by disabling the protocol or restricting UDP traffic.
• Response Mocking - Enables simulation of backend behaviors by replacing real server responses with custom status codes and bodies.
• Network Emulation Tools - Simulates offline mode and bandwidth throttling to test page performance under network constraints.
• Interactability Verifications - Verifies if an element is visible in the viewport and not blocked by overlays before interaction.
• Mouse Action Simulations - Dispatches clicks, movements, and wheel scrolling using viewport coordinates and button specifications.
• Shadow Root Traversals - Bypasses standard DOM restrictions to locate and interact with elements inside shadow roots.
• Network Traffic Monitors - Allows real-time observation of HTTP requests and responses through event callbacks and logs.
• Complex Interaction Synthesis - Generates high-level interactions such as pinch-to-zoom and multi-tap sequences.
• File Chooser Interception - Automates file uploads by intercepting the browser's native file chooser dialog.
• DOM Manipulators - Allows programmatic modification of the page by adding, removing, or duplicating DOM nodes.
• Content Manipulation - Enables programmatic replacement of the HTML content within specific frames.
• Drag and Drop Simulations - Implements programmatic simulation of dragging objects across screen coordinates to mimic human interaction.
• PDF and HTML Content Extraction - Retrieves visible text content from HTML strings with options to strip unnecessary whitespace.
• Layout Metric Extraction - Extracts detailed layout metrics, including dimensions and offsets, for elements on the page.
• Automatic Dialog Dismissers - Automatically detects and dismisses JavaScript alert and confirm dialogs to prevent automation deadlock.
• Key Sequence Simulators - Dispatches key presses and releases with support for modifier keys and physical key identifiers.
• Imperative Scroll Controllers - Provides imperative control over viewport movement to specific positions and distance calculations.
• Locale Spoofing - Configures the Accept-Language HTTP header and navigator language properties to match a target locale.
• Profile Seeding - Provides the ability to seed browser profiles with historical data to make fresh sessions appear established to anti-bot systems.
• DOM Structural Tracking - Monitors the DOM for structural mutations and attribute changes to trigger automation responses.
• Browser Argument Configuration - Customizes the browser instance using command-line arguments for headless mode and window dimensions.
• Browser Execution Isolations - Executes automation logic in isolated iframe environments to prevent conflicts with page scripts.
• Existing Tab Attachments - Allows connecting to and controlling browser tabs created by the user.
• Browser Resource Optimization - Optimizes execution speed by disabling resource-intensive features such as image loading and network prediction.
• Script Injections - Injects custom JavaScript to be evaluated automatically whenever a new document is created.
• Data Extraction - Restricts data recovery to specific page regions to isolate items for efficient bulk extraction.
• Element Wait Conditions - Pauses execution until specific elements become visible or interactable within a timeout.
• Retry and Backoff Logic - Ships a function wrapper that handles execution exceptions using exponential backoff for unstable environments.
• IFrame Context Resolvers - Automatically resolves and switches execution contexts to interact with elements nested in multiple iframes.
• Page Lifecycle Controls - Provides programmatic page reloading with options to bypass cache or inject scripts upon refresh.
• Resource Blocking - Prevents specific resource types like images or stylesheets from loading to reduce bandwidth usage.
• Element-Bound Scripts - Executes custom JavaScript where the script references the target element as the execution context.
• User Agent Parsers - Analyzes user agent strings to extract metadata used for synchronizing HTTP headers with browser properties.
• Browser Dialog Handlers - Programmatically responds to JavaScript alerts, confirmations, and prompts by accepting or dismissing them.
• Browser Permission Management - Programmatically grants or resets site permissions like geolocation to bypass manual prompts.
• File Upload Utilities - Automates file uploads by interacting with input elements or intercepting the file chooser.
• Full Page Screenshots - Captures visual representations of the visible viewport or full-length scrollable pages as files or base64 strings.

Star history