Awesome GitHub RepositoriesWeb Crawling Frameworks

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

This project is a self-hosted system for discovering, browsing, and receiving personalized recommendations from academic papers on arXiv. It combines an arXiv API client that downloads paper metadata and PDFs with a TF-IDF document similarity engine and an SVM-based recommendation system that trains a classifier per user based on their preferences. The system provides a web interface for browsing, searching, and filtering recent arXiv submissions, alongside personalized paper recommendations generated from individual user signals. It also includes a Twitter mention tracker that periodically p

This project is a distributed web crawling framework that enables the horizontal scaling of scraping tasks. It uses Redis as a centralized request queue manager and state store to coordinate crawl progress and request metadata across multiple server instances. The system distributes crawling workloads by sharing a single request queue and utilizes a distributed duplicate filter to prevent multiple workers from visiting the same page. It persists complex request state and metadata as JSON strings within the shared remote store. The framework also provides capabilities for distributed data pro

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

Crawler4j ist ein Multi-Threaded-Java-Webcrawler und -Spider für hochvolumiges Web-Traversing und Content-Extraktion. Es fungiert als „höfliches“ Crawling-Framework, das die Entdeckung und Indizierung von HTML- und Binärinhalten über mehrere Websites hinweg ermöglicht. Das Projekt zeichnet sich durch ein persistentes Crawling-Modell aus, das den Session-Status im lokalen Speicher serialisiert, sodass die Engine die Indizierung nach einem Absturz oder einer Unterbrechung fortsetzen kann. Es enthält einen Politeness-Controller zur Regulierung der Anfragefrequenz und -verzögerungen, um Serverüberlastungen und IP-Sperren zu vermeiden. Das System deckt eine breite Palette an Traversierungsfunktionen ab, einschließlich tiefenbegrenztem Scope-Management, Zielfilterung und Request-Interception für benutzerdefinierte User-Agents und Proxy-Routing. Die Datenspeicherung erfolgt über ein Repository-Pattern, das die Crawling-Logik von der Persistenz der Seitenmetadaten in relationalen Datenbanken entkoppelt.

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites. The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem. The system utilizes a pipeline-based data

MNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS

X-crawl ist ein auf Node.js basierendes Web-Scraping-Framework zur Automatisierung der Datenerfassung von statischen und dynamischen Websites. Es integriert KI für semantisches Parsing, um unstrukturiertes HTML in strukturierte Datenformate zu überführen, die auch bei Änderungen an Website-Layouts oder Klassennamen präzise bleiben. Das Projekt zeichnet sich durch eine umfassende Suite an Stealth- und Zuverlässigkeitsfunktionen aus. Es verwaltet die Crawler-Identität durch Randomisierung von Geräte-Fingerprints und Rotation von Proxy-Servern, um Zugriffsbeschränkungen zu umgehen. Für komplexe, JavaScript-lastige Interfaces nutzt es Headless-Browser-Automatisierung, um menschliches Verhalten wie Klicken und Tippen zu simulieren und so versteckte Inhalte während der Extraktion zugänglich zu machen. Das Framework bietet weitreichende Kontrolle über Crawling-Workflows durch Aufgabenplanung, Request-Priorisierung und Concurrency-Management. Integrierte Resilienz-Mechanismen wie automatische Retry-Logik und Fehlerbehandlung sorgen für konsistente Performance unter variierenden Netzwerkbedingungen. Zudem unterstützt es Lifecycle-Hooks für Dateidownloads und programmatisches Fortschritts-Monitoring zur Überwachung von Aufgaben und Ergebnissen.