awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
xianhu avatar

xianhu/PSpider

0
View on GitHub↗
1,840 stars·495 forks·Python·BSD-2-Clause·4 viewsgithub.com/xianhu/PSpider↗

PSpider

A simple web spider frame written by Python, which needs Python3.8+

Features

  • Python Crawling Frameworks - Simple spider framework for Python 3.

Star history

Star history chart for xianhu/pspiderStar history chart for xianhu/pspider

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Open-source alternatives to PSpider

Similar open-source projects, ranked by how many features they share with PSpider.
  • chineking/colachineking avatar

    chineking/cola

    1,501View on GitHub↗

    A high-level distributed crawling framework.

    Python
    View on GitHub↗1,501
  • cocrawler/cocrawlercocrawler avatar

    cocrawler/cocrawler

    194View on GitHub↗

    CoCrawler is a versatile web crawler built using modern tools and concurrency.

    Python
    View on GitHub↗194
  • codelucas/newspapercodelucas avatar

    codelucas/newspaper

    14,982View on GitHub↗

    Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

    HTMLcrawlercrawlingnews
    View on GitHub↗14,982
  • binux/pyspiderbinux avatar

    binux/pyspider

    16,809View on GitHub↗

    PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

    Python
    View on GitHub↗16,809
See all 21 alternatives to PSpider→

Frequently asked questions

What does xianhu/pspider do?

A simple web spider frame written by Python, which needs Python3.8+

What are the main features of xianhu/pspider?

The main features of xianhu/pspider are: Python Crawling Frameworks.

What are some open-source alternatives to xianhu/pspider?

Open-source alternatives to xianhu/pspider include: chineking/cola — A high-level distributed crawling framework. cocrawler/cocrawler — CoCrawler is a versatile web crawler built using modern tools and concurrency. codelucas/newspaper — Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a… douban/brownant — |Build Status| |Coverage Status| |PyPI Version| |PyPI Downloads| |Wheel Status|. gaojiuli/gain — Taken Over By Shad0w For Responsible Disclosure [Kiwi BBP]. binux/pyspider — PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for…