How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
The main features of stewartmckee/cobweb are: Ruby Crawling Frameworks.
Projects with overlapping indexed features include: felipecsl/wombat — Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. joenorton/rubyretriever — [RubyRetriever] (http://softwarebyjoe.com/rubyretriever/). postmodern/spidr — A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is… propublica/upton — A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.). sparklemotion/mechanize — Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically… sparklemotion/nokogiri — Nokogiri is an XML and HTML parsing library that builds navigable document trees from strings, files, or URLs using…
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
RubyRetriever (http://softwarebyjoe.com/rubyretriever/)
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)