Awesome GitHub RepositoriesPython Crawling Frameworks

Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

This project is a command-line utility designed to fetch video, audio, and image content from a wide range of web platforms. It functions by parsing page metadata and utilizing modular, site-specific scripts to extract direct media stream URLs from complex web structures, enabling the local archiving of digital media for offline use. The tool distinguishes itself through its ability to handle authenticated content, allowing users to inject browser-stored session cookies to access restricted or private media. It also supports real-time media streaming by piping remote content directly into ext

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web platforms, social media, and public repositories. It functions as a crawler framework that gathers proxy data and subscription links, a validation tool for testing server liveness, and a synchronization service for distributing the results. The system uses a plugin-based architecture that allows for the integration of custom Python scripts to handle diverse web source structures. It also includes utilities to transform raw proxy data into standardized configuration formats compatible w

QOwnNotes is a desktop note editor that stores each note as a plain-text Markdown file on the local filesystem, avoiding proprietary formats and enabling direct file access. It functions as a Nextcloud Notes client, syncing notes and metadata with Nextcloud or ownCloud servers through a companion API service for versioning and sharing. The application also integrates with AI providers and exposes a local MCP server for external agents to search and fetch notes, and includes a companion browser extension for capturing web content, bookmarks, and screenshots. The editor distinguishes itself thr

Ce projet est un framework de crawling web distribué qui permet la mise à l'échelle horizontale des tâches de scraping. Il utilise Redis comme gestionnaire de file d'attente de requêtes centralisé et magasin d'état pour coordonner la progression du crawl et les métadonnées des requêtes sur plusieurs instances de serveur. Le système distribue les charges de travail de crawling en partageant une file d'attente de requêtes unique et utilise un filtre de doublons distribué pour empêcher plusieurs workers de visiter la même page. Il persiste l'état complexe des requêtes et les métadonnées sous forme de chaînes JSON au sein du magasin distant partagé. Le framework fournit également des capacités pour le traitement de données distribué en poussant les éléments scrapés dans une file d'attente partagée pour une consommation parallèle par des workers de traitement séparés.

MechanicalSoup est une bibliothèque d'automatisation web Python conçue pour simuler le comportement d'un navigateur. Elle fonctionne comme une boîte à outils pour le web scraping et l'automatisation, fournissant un moteur d'analyse HTML et un gestionnaire de session HTTP pour interagir avec les sites web par programmation. La bibliothèque permet une interaction web headless en imitant une session utilisateur réelle. Elle gère l'état persistant via la gestion des cookies et le suivi automatique des redirections, permettant la navigation programmatique sur les sites web et la simulation d'interactions complexes avec le navigateur. Ses capacités couvrent le remplissage et la soumission automatisés de formulaires en utilisant des sélecteurs CSS, ainsi que l'extraction de données à partir de réponses HTML. L'ensemble d'outils inclut des utilitaires pour télécharger des fichiers liés, spécifier des agents utilisateur personnalisés et rechercher des pages basées sur des mots-clés spécifiques. Elle fournit également des outils de diagnostic pour rendre l'état actuel de la page dans un navigateur pour une vérification visuelle.

Robobrowser is a Python web scraping library that provides a headless browser emulator and an HTML DOM parser. It is designed to programmatically navigate websites, interact with HTML forms, and extract data from web pages. The tool includes a web request caching mechanism to store previously fetched web content, reducing network traffic and increasing loading speeds for repeated requests. It covers capabilities for automated web navigation, programmatic web scraping, and web form automation, including the ability to populate input fields and trigger submission events. The system also manage

Scrapely

A simple web spider frame written by Python, which needs Python3.8+

Ruia 🕸️ Async Python 3.6+ web scraping micro-framework based on asyncio. ⚡ Write less, run faster.

A high-level distributed crawling framework.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Minimalist and powerful Web Crawler.

Spidy (/spˈɪdi/) is the simple, easy to use command line web crawler. Given a list of web links, it uses the Python requests library to query the webpages. Spidy then uses lxml to extract all links from the page and adds them to its list. Pretty simple!

The information security department of 360 company has been recruiting for a long time and is interested in contacting the mailbox zhangxin1at360.cn.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

High Speed WebCrawler built on Eventlet. Supports databases engines like Postgre, Mysql, Oracle, Sqlite. Command line tools. Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python). Cookie Handlers. Very easy to use (see the example).