mishushakov/llm-scraper
Llm Scraper
Features
- LLM-Powered Scrapers - A tool that uses a language model to extract structured data from web pages based on a user-defined schema.
- Schema-Driven Extraction - Defines the shape of data to extract from webpages with Zod or JSON schemas.
- Webpage Parsers - Uses a language model to parse raw webpage content and return structured data matching a predefined schema.
- Structured Data Extraction - Parse webpage content with a language model to return fields defined by a Zod or JSON schema.
- LLM-Powered Webpage Parsers - Uses an LLM to parse a page's content and return typed fields defined by a Zod or JSON Schema.
- Headless Browser Automation - Leverages Playwright to programmatically control a browser for automated web content extraction.
- LLM-Powered Scrapers - Extracts structured data from web pages using a language model and a user-defined schema.
- Incremental Result Streaming - Yields extracted objects incrementally as the language model generates them for early consumption.
- Extraction Streams - Receives partial structured data incrementally as the language model processes a webpage.
- Provider-Agnostic Interfaces - Designed to work with various language models, allowing users to choose the provider or model.
- Zod Schemas - Accepts Zod schemas directly to define the structure and validation rules for extracted data fields.
- Schema Definitions - Supports JSON Schema as an alternative to Zod for defining extraction schemas.
- Incremental Result Streams - Yield partial structured results incrementally as the language model produces them for early consumption.
- Incremental Streams - Yields partial structured results incrementally as the language model produces them during extraction.
- Playwright Scripts - Produces a standalone Playwright script that replicates the extraction logic without requiring an LLM at runtime.
- Playwright Script Generators - Produces a Playwright script that extracts the same schema-defined data without an LLM call.
- Playwright Scripts - Creates reusable Playwright scripts that extract data without further language model calls.
