How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:
The main features of my8100/scrapydweb are: Web Scraping.
Projects with overlapping indexed features include: alir3z4/html2text — Convert HTML to Markdown-formatted text. alirezamika/autoscraper — Autoscraper is an automatic web scraping library and pattern-based data extractor that learns extraction rules from… antivanov/js-crawler — js-crawler. bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… binux/pyspider — PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for… adbar/trafilatura — Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web…
Convert HTML to Markdown-formatted text.
Autoscraper is an automatic web scraping library and pattern-based data extractor that learns extraction rules from sample data. It identifies and retrieves text, URLs, and HTML elements from web pages by analyzing sample values to replicate data patterns across different URLs. The system functions as a web scraping model manager, allowing users to save and reload learned rules to maintain consistent data extraction. It supports the export and import of scraping rules to a local file system to avoid repeating the training process for the same website. The library covers automated web data ex
Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high
