alirezamika/autoscraper
Autoscraper
Autoscraper is an automatic web scraping library and pattern-based data extractor that learns extraction rules from sample data. It identifies and retrieves text, URLs, and HTML elements from web pages by analyzing sample values to replicate data patterns across different URLs.
The system functions as a web scraping model manager, allowing users to save and reload learned rules to maintain consistent data extraction. It supports the export and import of scraping rules to a local file system to avoid repeating the training process for the same website.
The library covers automated web data extraction and web content harvesting through sample-based pattern learning and positional element retrieval. It can retrieve both specific data points and all elements on a page that match patterns identified from initial sample data.
Features
- Web Data Extraction - Provides a library for programmatically scraping and processing web content using learned patterns.
- Automated Web Scraping - Extracts structured information from websites by learning patterns from samples instead of manual selectors.
- Extraction Rule Learning - Analyzes sample text, URLs, or HTML tags to determine the patterns needed to retrieve similar web elements.
- Rule Derivation - Derives extraction rules by analyzing shared structural characteristics of provided text, URLs, or HTML tags.
- Pattern-Based Scraping - Identifies recurring elements on a webpage to automatically capture data across multiple similar URLs.
- Pattern-Based Extraction - Analyzes sample values to identify and replicate data patterns across different URLs.
- Web Content Harvesting - Collects structured data from HTML pages by training a model on examples of the desired output.
- Rule Serializers - Provides a mechanism to save learned extraction rules to the local file system for reuse across sessions.
- Model Persistence - Saves and retrieves learned scraping model files from disk to avoid repeating the training process.
- Pattern Extraction Utilities - Applies learned patterns to retrieve all elements on a page that match the training data.
- Scraping Configuration Portability - Allows writing learned extraction patterns to a file and reloading them to apply rules to different URLs.
- HTML Element Scrapers - Fetches specific or similar HTML elements based on learned positional and content-based rules.
- Scraping Model Managers - Saves and reloads learned scraping rules to maintain consistent data extraction across multiple pages.
- Extraction Rule Sets - Utilizes serialized extraction rule sets to parse and retrieve data from external web sources.
- Positional Element Access - Fetches specific data points by matching the exact index and order of elements found in training samples.
- Heuristic Selectors - Implements heuristic-based identification of UI elements to generate repeatable selectors for varying page layouts.
- Positional Data Extraction - Retrieves specific data points from pages based on the order and position of original samples.
- Structured Element Retrievers - Fetches multiple exact data points from a page using a predefined order based on sample values.
- Web Scraping - Automatic and lightweight web scraper.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
