# boris-code/feapder

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/boris-code-feapder).**

3,709 stars · 542 forks · Python · NOASSERTION

## Links

- GitHub: https://github.com/Boris-code/feapder
- Homepage: http://feapder.com
- awesome-repositories: https://awesome-repositories.com/repository/boris-code-feapder.md

## Topics

`crawler` `feapder` `feaplat` `python` `scrapy` `spider`

## Description

Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a distributed spider engine and a headless browser renderer to execute JavaScript and extract content from dynamic web pages.

The system includes a scalable data deduplicator to filter duplicate URLs and records during large-scale operations. A crawler monitoring system tracks the health of active scraping jobs and triggers alerts when system anomalies occur.

The framework provides capabilities for task scheduling, web data extraction, and resilient workflows that allow crawling tasks to resume from the last successful checkpoint after an interruption.

## Tags

### Data & Databases

- [Web Data Extraction](https://awesome-repositories.com/f/data-databases/web-data-extraction.md) — Provides programmatic tools for scraping and processing web content using specialized scrapers. ([source](https://cdn.jsdelivr.net/gh/boris-code/feapder@master/README.md))
- [Large-Scale Deduplications](https://awesome-repositories.com/f/data-databases/intra-dataset-deduplication/large-scale-deduplications.md) — Removes duplicate URLs and records from massive datasets using a scalable filtering mechanism.

### Content Management & Publishing

- [Dynamic Content Scraping](https://awesome-repositories.com/f/content-management-publishing/web-content-scraping/dynamic-content-scraping.md) — Extracts data from websites that require JavaScript rendering to load content before parsing.
- [JavaScript Rendering](https://awesome-repositories.com/f/content-management-publishing/web-page-scraping-extractors/javascript-rendering.md) — Executes JavaScript via headless browsers to load dynamic content before extracting final data. ([source](https://cdn.jsdelivr.net/gh/boris-code/feapder@master/README.md))

### Software Engineering & Architecture

- [Distributed Crawling Engines](https://awesome-repositories.com/f/software-engineering-architecture/distributed-systems/distributed-crawling-engines.md) — Provides a scalable architecture for managing multiple spider types and high-volume data collection.

### Web Development

- [Crawl Request Deduplications](https://awesome-repositories.com/f/web-development/api-request-deduplication/crawl-request-deduplications.md) — Implements a high-performance mechanism to filter duplicate URLs across distributed spiders.
- [Headless Browsers](https://awesome-repositories.com/f/web-development/headless-browsers.md) — Uses automated browser engines to render dynamic pages and execute scripts for data extraction.
- [Headless Rendering Engines](https://awesome-repositories.com/f/web-development/headless-browsers/headless-rendering-engines.md) — Integrates headless rendering engines to execute JavaScript and load dynamic content before parsing.
- [Web Crawling](https://awesome-repositories.com/f/web-development/web-automation-scraping/web-scraping-automation/web-crawling.md) — Manages high-volume data collection across domains with built-in deduplication and scheduling.
- [Web Crawling Frameworks](https://awesome-repositories.com/f/web-development/web-crawling-frameworks.md) — Comprehensive Python framework for automating web data extraction including scheduling and rendering.
- [Crawl Task Managers](https://awesome-repositories.com/f/web-development/task-execution-engines/crawl-task-managers.md) — Allows restarting crawling tasks from the last successful point after a failure or shutdown. ([source](https://cdn.jsdelivr.net/gh/boris-code/feapder@master/README.md))
- [Crawler Health Monitoring](https://awesome-repositories.com/f/web-development/web-automation-scraping/web-scraping-automation/web-scraping/crawler-health-monitoring.md) — Tracks the operational status and health of active scraping jobs with automated alerts.

### Part of an Awesome List

- [Spider Architectures](https://awesome-repositories.com/f/awesome-lists/devtools/web-and-html-processing/web-spiders/spider-architectures.md) — Implements distinct crawler classes tailored for single-page, task-based, or batch-processing extraction scenarios.

### DevOps & Infrastructure

- [Crawl Job Schedulers](https://awesome-repositories.com/f/devops-infrastructure/job-priority-management/priority-based-job-schedulers/scheduled-job-execution/crawl-job-schedulers.md) — Provides a centralized system to manage the deployment and execution timing of multiple scraping jobs.

### System Administration & Monitoring

- [Event-Driven Notification Triggers](https://awesome-repositories.com/f/system-administration-monitoring/alert-notification-systems/event-driven-notification-triggers.md) — Triggers real-time alerts based on failure thresholds or system anomalies during crawling.
- [Execution Progress Tracking](https://awesome-repositories.com/f/system-administration-monitoring/execution-progress-tracking.md) — Records task checkpoints in a persistent store to allow crawlers to resume from the last successful page.