mozilla/readability
Readability
Readability is a JavaScript library designed for web content extraction. It functions as a DOM parsing utility and article metadata extractor that isolates the primary text of a webpage by removing clutter such as advertisements and navigation bars.
The library employs a heuristic-based content detector to predict if a webpage contains a parseable article before performing full extraction. It uses a parsing workflow to convert complex HTML documents into a simplified format, facilitating the implementation of distraction-free reader views.
The tool covers several capability areas, including the identification of readable content, the extraction of the main article body, and the isolation of canonical titles and author information. It utilizes structural analysis and text filtering to distinguish primary content from sidebars and other non-essential elements.
Features
- Body Content Extractors - Parses HTML to isolate the primary body text while stripping markup and navigation elements.
- Full-Text Content Extraction - Processes web pages to retrieve a cleaned, full-text version of the article content.
- Extractability Detection - Evaluate a document using lightweight heuristics to determine if it contains an article suitable for extraction before starting full processing.
- Heuristic Extraction Methods - Analyzes HTML tag frequency, text density, and structural patterns to locate main content.
- Web Article Extraction - Isolates the main body text and core content from web pages and news articles.
- Metadata Extraction - Extracts descriptive page metadata, including Open Graph tags and canonical titles, from the document head.
- Clutter Removal - Strips non-essential elements like ads and navigation bars using predefined clutter patterns to isolate primary content.
- Reader Views - Creates simplified page presentations that strip away non-essential elements for focused reading.
- Content-Aware DOM Parsers - Analyzes document structure and heuristics to identify and extract readable content from webpages.
- DOM Density Analysis - Analyzes the DOM tree to identify parent containers with high densities of text and paragraph tags.
- Element Scoring Heuristics - Assigns numerical values to HTML elements based on class names and IDs to identify primary content areas.
- Pre-Extraction Heuristics - Evaluates document length and structure using simple triggers to avoid processing non-article pages.
- Web Article Metadata Extractors - Identifies and returns the title and metadata of a webpage for simplified reading views.
- Extractability Tests - Analyzes HTML structure to determine if a page contains a main body of text suitable for extraction.
- Text-to-Markup Ratios - Implements density-based filtering to distinguish article bodies from sidebars by analyzing text-to-markup ratios.
- Extraction Feasibility Analysis - Analyzes the structure of a document to determine if it contains extractable content before performing a full parse.
- Parsing Tools - Content extraction library.
