postlight/parser
Parser
Postlight Parser is a command-line tool that extracts the main article content from any web page URL, returning clean structured data including the title, author, date, excerpt, and lead image while stripping away ads and clutter. It uses a readability-based heuristic that scores HTML elements on text density and structural cues to identify the article body, and can accept pre-fetched HTML strings directly for parsing instead of fetching the URL.
The tool distinguishes itself through a modular architecture that supports domain-specific extractor overrides, allowing custom JavaScript modules to be loaded at runtime for particular domains to replace the generic extraction routine. It also enables CSS selector field injection, letting users extend parsed results with additional custom data by specifying selectors for single or multiple matches via CLI flags. Custom HTTP headers can be attached to every page request for authentication or site-specific requirements.
The parser offers a multi-format output pipeline that converts extracted HTML content into Markdown or plain text through a post-processing step before returning the result. All extraction functionality is exposed through a command-line interface that accepts URL, format, and header flags without requiring any code.
Features
- Web Article Extraction - Extracts main article content, title, author, and metadata from any web page URL while stripping ads and clutter.
- Heuristic Extraction Methods - Identifies main content by scoring HTML elements on text density, link ratio, and structural cues to isolate the article body.
- HTML Parsing Command Line Tools - Runs the extraction from a terminal and outputs the structured result directly to stdout.
- Web Content Parser CLI - Ships a terminal-based tool for retrieving and parsing web page content with format and header flags.
- Command Line Interface Design - Exposes all extraction functionality through a command-line tool that accepts URL, format, and header flags.
- Custom Extractor Implementations - Applies a user-defined extractor script to a URL during parsing without modifying the library's source code.
- Web-to-Markdown Conversions - Returns the extracted article body as GitHub-flavored Markdown instead of raw HTML.
- Markdown to Plain Text Converters - Returns the extracted article body as GitHub-flavored Markdown or plain text instead of the default HTML.
- Multi-Format Output Converters - Returns extracted article body as HTML, Markdown, or plain text based on a user-specified output format.
- CSS Selector - Adds user-defined fields to parsed output by specifying CSS selectors for single or multiple matches.
- Extractors - Loads custom JavaScript modules at runtime for particular domains to replace the generic extraction routine.
- Site-Specific Extractors - Registers a custom extraction script for a specific domain at runtime without modifying the library's source code.
- CSS Selector Data Extractors - Extends parsed results with additional fields by evaluating user-supplied CSS selectors against the page DOM.
- Customizable HTML Parsers - Accepts pre-fetched HTML strings and custom site extractors loaded at runtime for domain-specific parsing.
- CSS Selector Field Injections - Extends the parsed result with custom data by specifying CSS selectors for single or multiple matches via CLI flags.
