How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
The main features of propublica/upton are: Web Crawling, Ruby Crawling Frameworks.
Projects with overlapping indexed features include: felipecsl/wombat — Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. sparklemotion/mechanize — Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically… postmodern/spidr — A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is… lorien/web-scraping — This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools… mendableai/firecrawl-mcp-server — This project is a Model Context Protocol server that connects large language models to web scraping and crawling… adithya-s-k/omniparse — Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and…
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically navigating websites and simulating human behavior without a graphical user interface. The library provides an automated interface for populating and submitting web forms, including text fields, checkboxes, and file uploads. It manages stateful sessions by automatically storing and sending cookies across multiple requests to maintain user authentication and identity. Additional capabilities include web data scraping, the ability to download remote web content, and the maintena
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
This project is a Model Context Protocol server that connects large language models to web scraping and crawling tools. It functions as a bridge, allowing LLM clients to utilize a web crawling engine and scraping utilities to extract and process web data. The server integrates a markdown web converter that transforms dynamic web pages and PDF documents into clean markdown to optimize consumption by AI models. It also provides a browser automation interface for controlling headless sessions and bypassing access restrictions. The system covers broad capabilities including large-scale website d