jaypyles/ScraperrArchived
Scraperr
Scraperr 是一个自托管的 Web 抓取和爬虫平台，旨在通过 XPath 选择器从网站提取结构化数据。它作为一个容器化系统，通过队列管理抓取任务，并使用人工智能分析结果内容。
该项目的特色在于其 Kubernetes 原生架构，允许通过包管理器进行可扩展的部署和管理。它包括一个能够进行域名级爬取的爬虫引擎，以及一个使用人工智能查询提取出的 Web 内容的数据分析器。
该平台涵盖了广泛的功能，包括自动化数据提取、批量 Web 爬取和媒体文件下载。它提供了将抓取数据可视化为表格、配置自定义请求头以模拟浏览器身份，以及将结果导出为 CSV 或 Markdown 格式的工具。
该应用支持通过 Kubernetes 部署配置进行自定义安装参数和版本更新。
Features
- XPath Data Extractors - Uses XPath selectors to precisely target and extract specific data elements from web pages.
- Web Scraping and Extraction - Provides a self-hosted platform for extracting structured data from websites using XPath selectors.
- Automated Web Scraping - Provides a self-hosted platform for automatically extracting structured information from websites.
- CSS and XPath Query Engines - Utilizes XPath expressions to locate and extract specific nodes from parsed HTML documents.
- Domain-Restricted Crawling - Includes a crawling engine that limits link discovery to a specific root domain for focused data collection.
- Kubernetes Application Deployments - Utilizes a Kubernetes-native architecture for scalable deployment and automated application management.
- Job Queues - Manages bulk scraping tasks through a sequential job queue to ensure reliable data collection.
- Data Extractions - Uses XPath selectors and DOM traversal to pull structured data from multiple URLs.
- Domain Spidering - Implements a crawler that visits all linked pages within a domain to discover wide-ranging content.
- Scraping Infrastructure Management - Provides infrastructure to submit and track multiple scraping tasks via a centralized job queue.
- Web Crawling - Systematically discovers and indexes web content across entire domains for large-scale data collection.
- Web Scrapers - Ships an automated system for navigating websites and extracting structured data via a job queue.
- Selector Mappings - Enables precise data extraction by mapping target URLs to specific XPath selectors.
- Content Query Analyzers - Integrates AI to allow users to query and analyze the extracted web content using natural language.
- Web Content AI Analysis - Implements AI-driven analysis to query and retrieve information from extracted web content.
- AI-Powered Intelligence - Leverages artificial intelligence to query and extract insights from large volumes of scraped web content.
- CSV Data Exports - Exports scraped data into CSV and Markdown files for external analysis in spreadsheet applications.
- AI Data Analysis Tools - Provides a capability to answer questions based on scraped web data using AI model APIs.
- Kubernetes-Native Data Platforms - Implements a containerized scraping platform designed specifically for scalable orchestration on Kubernetes.
- File-Based Data Exports - Serializes collected web data into structured formats such as CSV and Markdown for external use.
- Scraped Data Exporters - Converts results from completed scraping jobs into structured CSV files.
- Kubernetes Application Deployments - Provides a Kubernetes-native architecture for scalable deployment and management of the scraping platform.
