# oxylabs/ai-crawler-py

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/oxylabs-ai-crawler-py).**

2,683 stars · 12 forks

## Links

- GitHub: https://github.com/oxylabs/ai-crawler-py
- Homepage: https://aistudio.oxylabs.io/apps/crawl
- awesome-repositories: https://awesome-repositories.com/repository/oxylabs-ai-crawler-py.md

## Topics

`ai` `ai-agents` `ai-crawler` `ai-scraping` `ai-studio` `ai-web-crawler` `crawl-agent` `web-crawler`

## Description

This project is an LLM-powered web crawler and data extractor that uses large language models to navigate websites and parse content into structured JSON or Markdown formats. It functions as an automated browser orchestrator and domain discovery engine, interpreting plain English instructions to identify relevant pages and extract specific information.

The system distinguishes itself through agentic browser automation, allowing it to perform human-like interactions such as clicking buttons and scrolling based on natural language commands. It employs goal-oriented crawling to analyze website structures and prioritize URL discovery according to high-level objectives rather than simple recursive linking.

The tool also includes capabilities for translating natural language requirements into search engine queries and generating OpenAPI schemas to enforce data contracts during extraction. Extracted data can be routed through a structured pipeline to external systems in real time via software development kits.

## Tags

### Data & Databases

- [LLM-Driven Data Extractors](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/data-parsing-extraction/llm-driven-data-extractors.md) — Uses large language models to transform unstructured web content into structured JSON or Markdown formats.
- [Data Parsing](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-extraction-ingestion/data-parsing.md) — Parses specific information from web pages into structured formats based on natural language descriptions. ([source](https://aistudio.oxylabs.io/))
- [Domain Structure Analyzers](https://awesome-repositories.com/f/data-databases/search-indexing-technologies/search-domains/domain-structure-analyzers.md) — Analyzes and maps website structures to intelligently identify and catalog relevant URLs for data collection. ([source](https://aistudio.oxylabs.io/))
- [Structured Data Extraction](https://awesome-repositories.com/f/data-databases/structured-data-extraction.md) — Extracts specific information from websites into structured JSON or Markdown formats using natural language prompts. ([source](https://cdn.jsdelivr.net/gh/oxylabs/ai-crawler-py@main/README.md))
- [Web Data Extraction](https://awesome-repositories.com/f/data-databases/web-data-extraction.md) — Programmatically scrapes and processes web content into structured formats using natural language prompts.
- [Data Extraction Pipelines](https://awesome-repositories.com/f/data-databases/data-extraction-pipelines.md) — Streams extracted web data into external systems through automated ingestion and synchronization pipelines.
- [OpenAPI Specification Enforcement](https://awesome-repositories.com/f/data-databases/schema-enforcement-tools/strict-schema-enforcers/openapi-specification-enforcement.md) — Generates OpenAPI specifications from text descriptions to ensure extracted data adheres to a strict contract.

### Web Development

- [AI-Powered Web Crawlers](https://awesome-repositories.com/f/web-development/web-automation-scraping/web-scraping-automation/web-scraping/ai-powered-web-crawlers.md) — Provides an LLM-powered web crawler that navigates pages and extracts structured data using natural language prompts.
- [Browser Interaction Actions](https://awesome-repositories.com/f/web-development/web-automation-scraping/browser-interaction-primitives/browser-interaction-actions.md) — Performs interactive browser operations like clicking and scrolling via natural language commands.
- [Web Crawling](https://awesome-repositories.com/f/web-development/web-automation-scraping/web-scraping-automation/web-crawling.md) — Systematically discovers and indexes web content across domains based on high-level goals and instructions.

### Artificial Intelligence & ML

- [AI-Powered Data Extractors](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-powered-data-extractors.md) — Leverages language models and OpenAPI schemas to transform unstructured web content into validated data.
- [Browser Automation Agents](https://awesome-repositories.com/f/artificial-intelligence-ml/browser-automation-agents.md) — Implements agents that interact with web browsers by interpreting natural language instructions for clicking and scrolling.
- [Intelligent Domain Mapping](https://awesome-repositories.com/f/artificial-intelligence-ml/intelligent-domain-mapping.md) — Analyzes website structures to intelligently identify and catalog important URLs for data collection.
- [AI-Powered Search](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-powered-search.md) — Translates natural language requirements into search queries to locate relevant information across the internet. ([source](https://aistudio.oxylabs.io/))
- [Natural Language Query Parsing](https://awesome-repositories.com/f/artificial-intelligence-ml/natural-language-query-parsing.md) — Translates human-readable requirements into specific search engine queries to locate relevant source pages.
- [Natural Language Schema Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/natural-language-schema-generation.md) — Uses AI to generate structured OpenAPI schemas from natural language descriptions of the desired data format. ([source](https://cdn.jsdelivr.net/gh/oxylabs/ai-crawler-py@main/README.md))

### Part of an Awesome List

- [Prompt-Guided Discovery](https://awesome-repositories.com/f/awesome-lists/devtools/web-crawling-and-scraping/prompt-guided-discovery.md) — Uses natural language prompts to guide the discovery and selection of relevant pages across a web domain. ([source](https://aistudio.oxylabs.io/))

### Development Tools & Productivity

- [Browser Automation Orchestrators](https://awesome-repositories.com/f/development-tools-productivity/browser-automation-orchestrators.md) — Coordinates headless browser engines to perform human-like interactions based on natural language instructions.
- [Browser Interactions](https://awesome-repositories.com/f/development-tools-productivity/natural-language-interfaces/browser-interactions.md) — Enables the manipulation of web elements and page navigation using plain English instructions. ([source](https://aistudio.oxylabs.io/))
- [Goal-Oriented Discovery Engines](https://awesome-repositories.com/f/development-tools-productivity/search-discovery-tools/recursive-discovery-engines/goal-oriented-discovery-engines.md) — Maps website structures and identifies relevant URLs to meet specific, high-level data goals.

### Software Engineering & Architecture

- [Goal-Oriented Crawling](https://awesome-repositories.com/f/software-engineering-architecture/crawl-prioritization-algorithms/goal-oriented-crawling.md) — Explores domains to find and prioritize specific types of pages based on user-defined goals. ([source](https://cdn.jsdelivr.net/gh/oxylabs/ai-crawler-py@main/README.md))
- [Goal-Oriented Discovery](https://awesome-repositories.com/f/software-engineering-architecture/crawl-prioritization-algorithms/goal-oriented-discovery.md) — Analyzes website structures to prioritize the discovery of pages that align with specific user-defined goals.