Awesome GitHub RepositoriesPython Crawling Frameworks

Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-based selectors. The system distinguishes itself through a highly modular architecture that supports complex data collection workflows. Users can implement custom middleware and signal handlers to intercept and modify request flows, while a priority-based scheduler manages concu

This project is a command-line utility designed to fetch video, audio, and image content from a wide range of web platforms. It functions by parsing page metadata and utilizing modular, site-specific scripts to extract direct media stream URLs from complex web structures, enabling the local archiving of digital media for offline use. The tool distinguishes itself through its ability to handle authenticated content, allowing users to inject browser-stored session cookies to access restricted or private media. It also supports real-time media streaming by piping remote content directly into ext

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web platforms, social media, and public repositories. It functions as a crawler framework that gathers proxy data and subscription links, a validation tool for testing server liveness, and a synchronization service for distributing the results. The system uses a plugin-based architecture that allows for the integration of custom Python scripts to handle diverse web source structures. It also includes utilities to transform raw proxy data into standardized configuration formats compatible w

QOwnNotes is a desktop note editor that stores each note as a plain-text Markdown file on the local filesystem, avoiding proprietary formats and enabling direct file access. It functions as a Nextcloud Notes client, syncing notes and metadata with Nextcloud or ownCloud servers through a companion API service for versioning and sharing. The application also integrates with AI providers and exposes a local MCP server for external agents to search and fetch notes, and includes a companion browser extension for capturing web content, bookmarks, and screenshots. The editor distinguishes itself thr

هذا المشروع عبارة عن إطار عمل موزع لكشط الويب يتيح التوسع الأفقي لمهام الكشط. يستخدم Redis كمدير طابور طلبات مركزي ومخزن حالة لتنسيق تقدم الكشط وبيانات تعريف الطلب عبر مثيلات خادم متعددة. يوزع النظام أعباء عمل الكشط من خلال مشاركة طابور طلبات واحد ويستخدم مرشح تكرار موزع لمنع العمال المتعددين من زيارة نفس الصفحة. ويحتفظ بحالة الطلب المعقدة وبيانات التعريف كسلاسل JSON داخل المخزن البعيد المشترك. يوفر إطار العمل أيضاً إمكانيات لمعالجة البيانات الموزعة عن طريق دفع العناصر المكتشطة إلى طابور مشترك للاستهلاك المتوازي بواسطة عمال معالجة منفصلين.

MechanicalSoup هي مكتبة أتمتة ويب Python مصممة لمحاكاة سلوك المتصفح. تعمل كمجموعة أدوات لكشط الويب والأتمتة، حيث توفر محرك تحليل HTML ومدير جلسة HTTP للتفاعل مع مواقع الويب برمجياً. تتيح المكتبة تفاعل الويب بدون رأس (headless) من خلال محاكاة جلسة مستخدم حقيقية. تدير الحالة المستمرة من خلال التعامل مع ملفات تعريف الارتباط ومتابعة إعادة التوجيه التلقائية، مما يسمح بالتنقل البرمجي في موقع الويب ومحاكاة تفاعلات المتصفح المعقدة. تغطي قدراتها ملء النماذج المؤتمتة وإرسالها باستخدام محددات CSS، بالإضافة إلى استخراج البيانات من استجابات HTML. تتضمن مجموعة الأدوات أدوات لتنزيل الملفات المرتبطة، وتحديد وكلاء مستخدم مخصصين، والبحث عن الصفحات بناءً على كلمات رئيسية محددة. كما توفر أدوات تشخيصية لعرض حالة الصفحة الحالية في متصفح للتحقق المرئي.

Robobrowser is a Python web scraping library that provides a headless browser emulator and an HTML DOM parser. It is designed to programmatically navigate websites, interact with HTML forms, and extract data from web pages. The tool includes a web request caching mechanism to store previously fetched web content, reducing network traffic and increasing loading speeds for repeated requests. It covers capabilities for automated web navigation, programmatic web scraping, and web form automation, including the ability to populate input fields and trigger submission events. The system also manage

Scrapely

A simple web spider frame written by Python, which needs Python3.8+

Ruia 🕸️ Async Python 3.6+ web scraping micro-framework based on asyncio. ⚡ Write less, run faster.

A high-level distributed crawling framework.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Minimalist and powerful Web Crawler.

Spidy (/spˈɪdi/) is the simple, easy to use command line web crawler. Given a list of web links, it uses the Python requests library to query the webpages. Spidy then uses lxml to extract all links from the page and adds them to its list. Pretty simple!

The information security department of 360 company has been recruiting for a long time and is interested in contacting the mailbox zhangxin1at360.cn.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

High Speed WebCrawler built on Eventlet. Supports databases engines like Postgre, Mysql, Oracle, Sqlite. Command line tools. Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python). Cookie Handlers. Very easy to use (see the example).