miso-belica/sumy
Sumy
Sumy is a text summarization library and toolset designed to identify and extract the most important sentences from plain text documents and HTML web pages. It functions as an extractive summarization system, meaning it selects representative original sentences from a source text without generating new words or modifying existing phrasing.
The project provides a suite of algorithmic methods for content distillation, including frequency-based sentence scoring and graph-based text ranking. It also includes utilities for HTML-to-text sanitization to isolate primary content and heuristic filtering to remove low-information noise.
Beyond extraction, the library includes a summarization evaluator to measure the accuracy and effectiveness of generated summaries. This is achieved by calculating precision and recall through word overlap between the output and a human-written reference text.
Features
- Text Summarization Tools - Provides a suite of tools for creating condensed versions of documents by identifying and ranking key sentences.
- Text Summarization - Extracts concise summaries from plain text and HTML pages using algorithmic methods.
- Keyword-Based Sentence Scoring - Implements importance scoring based on the density of high-frequency keywords relative to the document.
- Co-occurrence Ranking Algorithms - Calculates sentence importance by measuring connectivity and overlap within a weighted word co-occurrence graph.
- Extraction Accuracy Evaluators - Measures the precision and recall of extracted summaries against ground-truth corpora.
- Summarization Evaluation Tools - Provides a utility for assessing the factual accuracy and information coverage of generated summaries.
- Text Summary Metrics - Measures summary precision and recall by calculating word overlap between outputs and human-written gold standards.
- Stop Word Filters - Includes mechanisms for removing high-frequency, non-informative stop-words to improve summary quality.
- PDF and HTML Content Extraction - Extracts and condenses primary content from HTML web pages for more efficient reading.
- HTML to Text Conversion - Transforms HTML documents into plain text to isolate primary content for linguistic analysis.
- Content Extraction - Automatic summarization of text and HTML.
- Web Scraping - Automatic summarization for text and HTML.
