awesome-repositories.com
Blog
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectAboutHow we rankPressMCP server
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
jmg avatar

jmg/crawley

0
View on GitHub↗
191 stars·34 forks·Python·GPL-3.0·2 viewsproject.crawley-cloud.com↗

Crawley

High Speed WebCrawler built on Eventlet. Supports databases engines like Postgre, Mysql, Oracle, Sqlite. Command line tools. Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python). Cookie Handlers. Very easy to use (see the example).

Features

  • Python Crawling Frameworks - Pythonic framework based on non-blocking I/O.

Star history

Star history chart for jmg/crawleyStar history chart for jmg/crawley

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Open-source alternatives to Crawley

Similar open-source projects, ranked by how many features they share with Crawley.
  • chineking/colachineking avatar

    chineking/cola

    1,501View on GitHub↗

    A high-level distributed crawling framework.

    Python
    View on GitHub↗1,501
  • cocrawler/cocrawlercocrawler avatar

    cocrawler/cocrawler

    194View on GitHub↗

    CoCrawler is a versatile web crawler built using modern tools and concurrency.

    Python
    View on GitHub↗194
  • codelucas/newspapercodelucas avatar

    codelucas/newspaper

    14,982View on GitHub↗

    Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

    HTMLcrawlercrawlingnews
    View on GitHub↗14,982
  • binux/pyspiderbinux avatar

    binux/pyspider

    16,809View on GitHub↗

    PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

    Python
    View on GitHub↗16,809
See all 21 alternatives to Crawley→

Frequently asked questions

What does jmg/crawley do?

High Speed WebCrawler built on Eventlet. Supports databases engines like Postgre, Mysql, Oracle, Sqlite. Command line tools. Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python). Cookie Handlers. Very easy to use (see the example).

What are the main features of jmg/crawley?

The main features of jmg/crawley are: Python Crawling Frameworks.

What are some open-source alternatives to jmg/crawley?

Open-source alternatives to jmg/crawley include: chineking/cola — A high-level distributed crawling framework. cocrawler/cocrawler — CoCrawler is a versatile web crawler built using modern tools and concurrency. codelucas/newspaper — Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a… douban/brownant — |Build Status| |Coverage Status| |PyPI Version| |PyPI Downloads| |Wheel Status|. gaojiuli/gain — Taken Over By Shad0w For Responsible Disclosure [Kiwi BBP]. binux/pyspider — PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for…