Pyspider

Features

Distributed Crawling Systems - Provides a framework for high-volume, asynchronous web crawling across multiple nodes using message queues.
Web Crawling - Provides a configurable framework for systematically discovering and indexing web content across domains for large-scale data collection.
JavaScript Rendering - Retrieves data from JavaScript-heavy websites by rendering pages with a headless browser before extraction.
Data Extraction Pipelines - Implements a workflow for periodically fetching web content, processing HTML, and persisting data into databases.
Web Data Extraction - Automates the process of visiting websites and extracting specific information into structured formats.
Distributed Task Queues - Uses message queues to distribute crawling tasks across multiple worker nodes for increased throughput.
Multi-node Orchestration - Features a distributed architecture that scales data collection via a cluster of nodes and a central controller.
Distributed Crawl Coordination - Coordinates the partitioning and synchronization of web discovery tasks across multiple worker nodes.
Headless Browsers - Includes a headless browser to execute JavaScript and capture rendered HTML from dynamic web pages.
Headless Rendering Engines - Utilizes a headless rendering engine to execute JavaScript on dynamic web pages for scraping.
Crawl Task Managers - Implements a system for organizing web crawling jobs into projects to manage the lifecycle of individual crawl tasks.
Web Crawling Frameworks - A Python-based framework for automating data extraction from websites with built-in scheduling and management.
Management Interfaces - Offers a web-based user interface for controlling scraping scripts and monitoring the progress of data collection.
Data Persistence and Storage - Persists collected web information into various database backends for long-term storage and retrieval.
Task Scheduling - Implements recurring crawl job triggers based on defined time intervals or content age.
Task Coordinations - Provides mechanisms for synchronizing crawl workflows and tracking task completion across distributed worker nodes.
Database-Backed Persistence - Provides the ability to persist scraped data and task states in external databases for reliability and recovery.
Web Scraping and Automation - Automates browser interactions and crawl scheduling to keep collected data up to date.
Management Interfaces - Provides a visual interface for editing scraping scripts and monitoring data collection without manual configuration.
Web Scraping Management Interfaces - Provides a browser interface to write, edit, and deploy scraping scripts directly to the running system.
Python Crawling Frameworks - Powerful, full-featured spider system.
Python Frameworks and Tools - Web crawling system.
Web Scraping - Comprehensive web crawling system.

Open-source alternatives to Pyspider

Similar open-source projects, ranked by how many features they share with Pyspider.

apify/crawlee
apify/crawlee
24,002View on GitHub
Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
TypeScriptapifyautomationcrawler
View on GitHub24,002
scrapinghub/portia
scrapinghub/portia
9,509View on GitHub
Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex
Python
View on GitHub9,509
code4craft/webmagic
code4craft/webmagic
11,680View on GitHub
Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col
Javacrawlerframeworkjava
View on GitHub11,680
apify/crawlee-python
apify/crawlee-python
8,097View on GitHub
Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra
Pythonapifyautomationbeautifulsoup
View on GitHub8,097

See all 30 alternatives to Pyspider

binuxpyspiderArchived

Features

Open-source alternatives to Pyspider

apify/crawlee

scrapinghub/portia

code4craft/webmagic

apify/crawlee-python

Star history

Open-source alternatives to Pyspider

apify/crawlee

scrapinghub/portia

code4craft/webmagic

apify/crawlee-python