Awesome GitHub RepositoriesPython Crawling Frameworks

Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

This project is a command-line utility designed to fetch video, audio, and image content from a wide range of web platforms. It functions by parsing page metadata and utilizing modular, site-specific scripts to extract direct media stream URLs from complex web structures, enabling the local archiving of digital media for offline use. The tool distinguishes itself through its ability to handle authenticated content, allowing users to inject browser-stored session cookies to access restricted or private media. It also supports real-time media streaming by piping remote content directly into ext

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web platforms, social media, and public repositories. It functions as a crawler framework that gathers proxy data and subscription links, a validation tool for testing server liveness, and a synchronization service for distributing the results. The system uses a plugin-based architecture that allows for the integration of custom Python scripts to handle diverse web source structures. It also includes utilities to transform raw proxy data into standardized configuration formats compatible w

QOwnNotes is a desktop note editor that stores each note as a plain-text Markdown file on the local filesystem, avoiding proprietary formats and enabling direct file access. It functions as a Nextcloud Notes client, syncing notes and metadata with Nextcloud or ownCloud servers through a companion API service for versioning and sharing. The application also integrates with AI providers and exposes a local MCP server for external agents to search and fetch notes, and includes a companion browser extension for capturing web content, bookmarks, and screenshots. The editor distinguishes itself thr

This project is a distributed web crawling framework that enables the horizontal scaling of scraping tasks. It uses Redis as a centralized request queue manager and state store to coordinate crawl progress and request metadata across multiple server instances. The system distributes crawling workloads by sharing a single request queue and utilizes a distributed duplicate filter to prevent multiple workers from visiting the same page. It persists complex request state and metadata as JSON strings within the shared remote store. The framework also provides capabilities for distributed data pro

MechanicalSoup ist eine Python-Webautomatisierungsbibliothek, die dazu entwickelt wurde, Browserverhalten zu simulieren. Sie fungiert als Toolkit für Web-Scraping und Automatisierung und bietet eine HTML-Parsing-Engine sowie einen HTTP-Session-Manager, um programmatisch mit Websites zu interagieren. Die Bibliothek ermöglicht headless Web-Interaktion durch Nachahmung einer echten Benutzersitzung. Sie verwaltet den persistenten Zustand durch Cookie-Handling und automatisches Folgen von Redirects, was die programmatische Website-Navigation und die Simulation komplexer Browser-Interaktionen ermöglicht. Ihre Funktionen decken das automatisierte Ausfüllen und Absenden von Formularen unter Verwendung von CSS-Selektoren sowie die Datenextraktion aus HTML-Antworten ab. Das Toolset enthält Utilities zum Herunterladen verlinkter Dateien, zur Spezifikation benutzerdefinierter User-Agents und zur Suche nach Seiten basierend auf spezifischen Keywords. Es bietet zudem Diagnose-Tools, um den aktuellen Seitenstatus zur visuellen Überprüfung in einem Browser zu rendern.

Robobrowser is a Python web scraping library that provides a headless browser emulator and an HTML DOM parser. It is designed to programmatically navigate websites, interact with HTML forms, and extract data from web pages. The tool includes a web request caching mechanism to store previously fetched web content, reducing network traffic and increasing loading speeds for repeated requests. It covers capabilities for automated web navigation, programmatic web scraping, and web form automation, including the ability to populate input fields and trigger submission events. The system also manage

Scrapely

A simple web spider frame written by Python, which needs Python3.8+

Ruia 🕸️ Async Python 3.6+ web scraping micro-framework based on asyncio. ⚡ Write less, run faster.

A high-level distributed crawling framework.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Minimalist and powerful Web Crawler.

Spidy (/spˈɪdi/) is the simple, easy to use command line web crawler. Given a list of web links, it uses the Python requests library to query the webpages. Spidy then uses lxml to extract all links from the page and adds them to its list. Pretty simple!

The information security department of 360 company has been recruiting for a long time and is interested in contacting the mailbox zhangxin1at360.cn.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

High Speed WebCrawler built on Eventlet. Supports databases engines like Postgre, Mysql, Oracle, Sqlite. Command line tools. Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python). Cookie Handlers. Very easy to use (see the example).