Boris-code/feapder
Feapder
Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a distributed spider engine and a headless browser renderer to execute JavaScript and extract content from dynamic web pages.
The system includes a scalable data deduplicator to filter duplicate URLs and records during large-scale operations. A crawler monitoring system tracks the health of active scraping jobs and triggers alerts when system anomalies occur.
The framework provides capabilities for task scheduling, web data extraction, and resilient workflows that allow crawling tasks to resume from the last successful checkpoint after an interruption.
Features
- Web Data Extraction - Provides programmatic tools for scraping and processing web content using specialized scrapers.
- Dynamic Content Scraping - Extracts data from websites that require JavaScript rendering to load content before parsing.
- JavaScript Rendering - Executes JavaScript via headless browsers to load dynamic content before extracting final data.
- Large-Scale Deduplications - Removes duplicate URLs and records from massive datasets using a scalable filtering mechanism.
- Distributed Crawling Engines - Provides a scalable architecture for managing multiple spider types and high-volume data collection.
- Crawl Request Deduplications - Implements a high-performance mechanism to filter duplicate URLs across distributed spiders.
- Headless Browsers - Uses automated browser engines to render dynamic pages and execute scripts for data extraction.
- Headless Rendering Engines - Integrates headless rendering engines to execute JavaScript and load dynamic content before parsing.
- Web Crawling - Manages high-volume data collection across domains with built-in deduplication and scheduling.
- Web Crawling Frameworks - Comprehensive Python framework for automating web data extraction including scheduling and rendering.
- Spider Architectures - Implements distinct crawler classes tailored for single-page, task-based, or batch-processing extraction scenarios.
- Crawl Job Schedulers - Provides a centralized system to manage the deployment and execution timing of multiple scraping jobs.
- Event-Driven Notification Triggers - Triggers real-time alerts based on failure thresholds or system anomalies during crawling.
- Execution Progress Tracking - Records task checkpoints in a persistent store to allow crawlers to resume from the last successful page.
- Crawl Task Managers - Allows restarting crawling tasks from the last successful point after a failure or shutdown.
- Crawler Health Monitoring - Tracks the operational status and health of active scraping jobs with automated alerts.
