A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
الميزات الرئيسية لـ propublica/upton هي: Web Crawling, Ruby Crawling Frameworks.
تشمل البدائل مفتوحة المصدر لـ propublica/upton: felipecsl/wombat — Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. sparklemotion/mechanize — Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically… postmodern/spidr — A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is… lorien/web-scraping — This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools… mendableai/firecrawl-mcp-server — This project is a Model Context Protocol server that connects large language models to web scraping and crawling… adithya-s-k/omniparse — Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and…
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically navigating websites and simulating human behavior without a graphical user interface. The library provides an automated interface for populating and submitting web forms, including text fields, checkboxes, and file uploads. It manages stateful sessions by automatically storing and sending cookies across multiple requests to maintain user authentication and identity. Additional capabilities include web data scraping, the ability to download remote web content, and the maintena
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
This project is a Model Context Protocol server that connects large language models to web scraping and crawling tools. It functions as a bridge, allowing LLM clients to utilize a web crawling engine and scraping utilities to extract and process web data. The server integrates a markdown web converter that transforms dynamic web pages and PDF documents into clean markdown to optimize consumption by AI models. It also provides a browser automation interface for controlling headless sessions and bypassing access restrictions. The system covers broad capabilities including large-scale website d