Dev Browser

Dev-browser is a browser automation framework and headless browser controller that provides a sandboxed script runner for executing web tasks. It functions as a vision-based web automator and a specialized interface for large language models, enabling the navigation and interaction of web pages within isolated execution environments.

The project distinguishes itself by converting complex web pages into simplified representations and coordinate-based maps, allowing AI agents to analyze layouts and perform actions based on pixel locations. It employs a mapping system that assigns unique identifiers to DOM elements, decoupling interaction logic from volatile page selectors.

The system covers a broad range of automation capabilities, including persistent session and page management to maintain state across script executions, headless browser lifecycle control, and the generation of AI-friendly page snapshots for state analysis. It also includes security primitives to restrict script access to the host filesystem and network.

The framework is implemented using TypeScript and leverages Playwright for its programmable browser interface.

Features

Visual Automation Mappings - Provides a critical mapping system that translates visual page locations into precise coordinates for AI-driven interaction.

Headless Browser Controllers - Acts as a specialized controller for managing remote browser sessions and persisting page states across scripts.

Agentic Browser Interfaces - Converts complex web pages into coordinate-based maps and simplified representations for AI agents to analyze.

Agentic Web Interaction - Enables AI agents to navigate and interact with web content using coordinate maps and page summaries.

Simplified Page Layouts - Creates visual coordinate maps and document summaries for the automated perception of page layouts.

AI-Ready Page Summaries - Converts complex HTML into simplified spatial and textual representations optimized for AI agent consumption.

Browser Automation Frameworks - Provides a programmable framework leveraging Playwright to execute sandboxed scripts for web navigation and interaction.

Browser-Based Workflows - Provides a programmable interface for writing and running scripts to automate web browser interactions and workflows.

Sandboxed Browser Runtimes - Provides a secure, isolated environment for executing automation scripts with restricted filesystem and network access.

Coordinate-Based Interactions - Performs clicks and keyboard inputs based on precise pixel coordinates for vision-based interaction.

DOM Element ID Mapping - Decouples interaction logic from volatile CSS/XPath selectors by assigning unique IDs to DOM elements.

Browser Automation - Provides a programmable framework for executing scripts that navigate web pages and interact with elements.

Chrome DevTools Protocols - Implements low-level browser control by utilizing the Chrome DevTools Protocol for headless instance management.

Headless Browser Orchestrators - Provides a mechanism to launch headless browsers and attach to running instances for automated tasks.

Vision-Based Page Interactions - Performs clicks and keyboard inputs based on pixel coordinates and visual layout maps instead of DOM selectors.

AI-Optimized Page Snapshots - Generates simplified page representations using coordinate mapping to help AI agents interact with visual elements.

Sandboxed Scripting - Executes automation scripts within a restricted sandbox to ensure host system security.

Code Execution Sandboxes - Provides a secure, isolated environment that prevents automation scripts from accessing the host filesystem and network.

Stateful Session Persistence - Maintains browser context and authentication state across multiple script executions to avoid redundant navigation.

Virtual-to-DOM Mappings - Maps visible browser elements to unique identifiers to execute actions against specific page IDs.

Browser Page Management - Enables the maintenance of named pages and contexts across scripts to preserve session state.

Page State Analysis - Inspects and interprets the current DOM and browser state to create summaries for automated decision-making.

Browser Session Persistence - Maintains active browser sessions between separate script runs to bypass repetitive navigation steps.

Web Automation and Scraping - Captures page states and snapshots to enable automated layout analysis and data extraction.

Browser Lifecycle Managers - Manages the startup, shutdown, and connection stability of remote browser processes during automation.

SawyerHooddev-browser

Features

Star history