h4ckf0r0day/obscura
Obscura
Obscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for AI models to control browser sessions, interact with websites, and extract web data using a WebSocket implementation of the Chrome DevTools Protocol.
The project focuses on bot detection evasion by randomizing browser fingerprints, masking native functions, and blocking tracking scripts to mimic human behavior. It further secures identities through a traffic layer that routes network requests via HTTP or SOCKS5 proxies.
The system supports large-scale data extraction through concurrent worker execution and dynamic JavaScript rendering. To optimize data for large language models, it includes utilities that transform structured HTML into flattened markdown. Low-level engine configuration is also available via flags to tune memory allocation and runtime performance.
Features
- Agentic Web Browsing - Provides AI models with the ability to control browser sessions and interact with live websites.
- External API Tool Exposures - Exposes browser control and data extraction functions as callable tools for AI agents.
- JavaScript Rendering - Executes JavaScript via headless browsers to render dynamic content before extracting the final HTML.
- Concurrent Scraping Workers - Implements parallel worker execution to efficiently extract large volumes of data from multiple URLs.
- User Interaction Simulators - Simulates human web activity through programmatic clicks, form filling, and page navigation.
- Headless Browser Automation - Automates user actions and data extraction using a headless browser controlled via the Chrome DevTools Protocol.
- Anti-Bot Evasion - Uses browser fingerprint randomization and function masking to bypass automated bot detection services.
- Fingerprint Randomization - Randomizes browser fingerprints and masks native properties to evade automated bot detection systems.
- Headless Browser Controllers - Provides a control layer for managing headless browser instances that AI agents use to interact with the web.
- Remote Browser Controllers - Provides a remote control layer for managing browser instances and page states via standardized messaging.
- Chrome DevTools Protocols - Implements the Chrome DevTools Protocol to enable bidirectional communication and control of browser instances.
- Scraping Infrastructure Management - Manages the operational infrastructure for large-scale scraping, including proxy routing and bot evasion.
- Web Scraping Evasion Tools - Employs anti-detection techniques and fingerprint randomization to scrape websites without being flagged as a bot.
- Browser Automation WebSocket Bridges - Implements a WebSocket bridge that allows external automation libraries to control remote browser instances.
- HTML to Markdown Converters - Transforms complex HTML structures into token-efficient markdown for AI analysis.
- DOM-to-Markdown Transformations - Transforms structured HTML trees into flattened markdown to optimize token usage for large language models.
- Markdown Converters - Converts raw webpage content into clean markdown format specifically for consumption by large language models.
- Headless Browser Workers - Processes large volumes of URLs by executing multiple headless browser sessions in parallel across separate processes.
- Proxy Routing - Routes network requests through external HTTP or SOCKS5 proxies to mask requester identity.
