scrapinghub/portia
Portia
Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code.
The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in external environments.
The platform covers comprehensive web crawler management, including the ability to handle nested hierarchical data structures and monitor failed page tracking for crawl stability. It provides tools for scraping project management and extracted data retrieval, utilizing a headless browser to render pages for visual element selection.
The application is packaged for deployment via containerization to ensure consistent runtime environments.
Features
- Visual Web Scraping Tools - Provides a graphical point-and-click interface to define scraping logic without writing code.
- Visual Extraction Interfaces - Enables capturing structured information from web pages by annotating elements visually instead of writing scripts.
- Hierarchical Data Structures - Organizes extracted data using parent-child relationships to capture complex page layouts like lists containing detailed items.
- Hierarchical Extraction - Captures complex hierarchical data structures from web pages by nesting extracted items.
- Web Data Extraction - Provides capabilities to programmatically fetch and extract structured data from specific web URLs.
- Web Data Extraction Tools - Handles nested hierarchical structures and repetitive page patterns to extract detailed datasets from the web.
- Spider Logic Generators - Converts visual web page annotations into executable Scrapy spider code for use in external environments.
- Visual-to-Specification Mapping - Translates graphical UI annotations into structured JSON specifications interpreted as extraction rules by the scraping engine.
- Headless Browsers - Integrates a headless browser to render dynamic web pages and allow visual selection of elements from the DOM.
- Web Crawlers - Organizes multiple scraping projects and monitors page failures to ensure stable data collection workflows.
- Web Crawling Frameworks - Leverages the Scrapy framework for high-performance asynchronous request scheduling, concurrency, and page fetching.
- Web Scraping Management Interfaces - Ships a web-based interface for organizing, copying, and managing multiple scraping projects and spiders.
- Project Export Environments - Converts visually defined scraping projects into executable code for deployment in external environments.
- Container Deployment - Packages the application and its dependencies into standardized images for consistent deployment across environments.
- Container Orchestration Deployments - Coordinates the application and its dependencies using container definitions to ensure environment consistency.
- Containerized Scraping Platforms - Provides a web application packaged via Docker to manage and execute automated web crawls.
- Configuration-Driven Logic - Stores scraping instructions as JSON objects instead of executable scripts to facilitate dynamic modification and export.
- Container-Based Isolation - Utilizes container-based isolation to ensure consistent runtime environments for the scraping platform across different servers.
- Crawl Stability Monitoring - Monitors and blocks web pages that fail to load to maintain stable and efficient crawling workflows.
- Python Crawling Frameworks - Visual scraping tool for Scrapy.
- Web Scraping - Visual web scraping tool.
