GeneralNewsExtractor/GeneralNewsExtractor
GeneralNewsExtractor
GeneralNewsExtractor is a specialized system for identifying and extracting structured news data through configurable HTML filtering and XPath-based parsing. It functions as an automated news scraper and web page content parser that isolates article bodies and metadata from news webpages.
The system uses YAML or JSON configuration files to manage extraction rules and XPath definitions, decoupling the processing logic from site-specific selectors. This allows for targeted noise removal and the isolation of titles, authors, and publication dates through customizable filtering.
The tool covers a broad range of extraction capabilities, including index page parsing for identifying article lists, relative URL resolution for images, and sample-driven list parsing to capture repeating data structures. These utilities enable the automated archiving of clean news content by removing irrelevant HTML elements and boilerplate.
Features
- News and Blog Article Scraping - Provides a specialized system for scraping and isolating article bodies and metadata from news websites using XPath.
- Web Article Extraction - Isolates main body text and core news content from web pages by removing structural noise.
- XPath Data Extractors - Uses precise XPath rules and configuration files to target specific elements for consistent data collection.
- Automated News Scrapers - A data extraction system that identifies titles, authors, and dates from news sites via YAML or JSON configuration.
- Web Content Noise Reduction - Provides pre-processing capabilities to strip irrelevant boilerplate and noise from scraped news page HTML.
- Full-Text Content Extraction - Provides a comprehensive process to retrieve a cleaned, full-text version of news articles including metadata and images.
- CSS and XPath Query Engines - Uses XPath query engines to precisely locate and isolate specific text and metadata elements from HTML.
- HTML Noise Removal - Provides pre-processing capabilities to eliminate irrelevant HTML elements using blacklisted XPaths before extraction.
- Metadata Extraction - Extracts specific news metadata including titles, authors, and publication dates from HTML.
- Extraction Rule Sets - Manages collections of serialized logic in YAML or JSON to maintain consistent data extraction across different news sites.
- Custom Extraction Rule Definitions - Provides a system for defining and loading site-specific scraping rules using YAML files.
- HTML Boilerplate Removal - Strips non-essential layout elements like navigation and sidebars using XPath lists to clean content.
- Structured Content Extractors - Implements an HTML processing utility that extracts structured text and list data from news index pages.
- List Data Extraction - Identifies list structures based on sample elements or XPaths to return all items within a target list.
- Sample-Driven List Parsing - Identifies repeating data structures by analyzing a single sample element to extract all matching items from a page.
- Web Page Parsing - Parses news index pages to identify lists of articles and their associated links using XPaths.
- Multi-Stage Content Processing - Implements an architectural pattern for processing news content through sequential cleaning and extraction stages.
- Web List Element Capture - Captures all items within a webpage list by analyzing sample XPaths or content snippets from a single row.
