Awesome GitHub RepositoriesWeb Scraping

This project is an AI agent orchestration platform that provides a visual environment for building, testing, and deploying complex automation workflows. It functions as a low-code development interface where users can chain discrete functional blocks into dependency-aware pipelines to integrate artificial intelligence with external data and services. The platform supports the creation of intelligent conversational agents, automated business processes, and multi-service API orchestrations within a unified workspace. The platform distinguishes itself through its event-driven integration engine,

This project serves as a community-maintained repository of documentation for the internal application programming interfaces used by the Bilibili video platform. It provides a structured reference for developers to understand service endpoints, request parameters, and data structures that are not officially documented. The collection relies on reverse-engineered network traffic analysis to map how client applications communicate with backend infrastructure. By decoding these proprietary protocols, the project offers a unified interface that allows developers to build third-party clients, agg

This is a collection of Python scripts designed for extracting data from popular Chinese websites and mobile applications. It functions as a multi-platform data extraction toolkit, capable of automating tasks such as downloading videos from platforms like Bilibili and Douyin, scraping product reviews and images from e-commerce sites like Taobao and JD.com, and booking train tickets on the 12306 railway system. The project distinguishes itself through its focus on automating specific, high-value tasks within the Chinese internet ecosystem. It includes capabilities for solving Chinese CAPTCHA c

This repository is an educational collection of standalone Python scripts designed to help developers master foundational programming concepts. It serves as a practical learning resource that demonstrates common coding patterns through modular, independent programs. The project focuses on building utility scripts for tasks such as file manipulation, data conversion, web scraping, and system automation. By relying exclusively on the Python standard library, the collection ensures portability and minimizes the need for external environment configuration. The scripts follow a procedural control

node-lessons is a comprehensive Node.js programming course and instructional guide. It provides a collection of guided lessons and code examples designed to teach the fundamentals of the Node.js runtime and server-side JavaScript development. The project serves as a practical guide for building web servers and backend applications, specifically covering the implementation of HTTP servers, request routing, and middleware chains. It includes specialized instructional material on managing asynchronous JavaScript workflows through promises and flow control, as well as guides for integrating NoSQL

This project is a Python-based automation toolkit designed to manage programmatic authentication and session persistence across web services. It provides a framework for executing automated login sequences, including the handling of interactive security challenges such as QR code verification and captcha resolution. The toolkit distinguishes itself by simulating native mobile application environments, allowing for the execution of scripts that require specific device-level headers and behaviors. It also incorporates hook-based interception to monitor workflow states and manage exceptions duri

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

TikTokDownloader is a containerized automation tool designed for the systematic collection and archiving of social media content. It functions as an HTTP API-driven service that enables users to trigger automated media retrieval and metadata extraction tasks through standard web requests, facilitating the storage of videos, images, and platform data for offline analysis. The utility distinguishes itself through session-based web scraping, which utilizes browser cookies to authenticate requests and access restricted platform content that is otherwise unavailable. By maintaining active session

This project is a machine learning data pipeline designed to automate the collection, curation, and preparation of large-scale image datasets. It functions as an image dataset scraper and computer vision curator, providing the necessary infrastructure to aggregate categorized files from web sources and organize them into structured directories for model development. The system distinguishes itself through a batch-processing architecture that integrates data acquisition with automated integrity validation. By scanning files to remove corrupted or invalid images and applying deterministic parti

Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

Jsoup is a Java library designed for parsing, extracting, and manipulating HTML and XML content. It provides a document object model that represents web content as a hierarchical tree, allowing for programmatic navigation and modification of elements, attributes, and text. The library functions as a toolkit for web scraping, enabling the retrieval of remote content via standard web protocols and the management of HTTP sessions for automated form interaction. The library distinguishes itself through its fault-tolerant tokenization, which reconstructs valid document structures from malformed or

Grocy is a self-hosted household management system designed to track grocery inventory, chore logs, and equipment manuals within a private home environment. It functions as a grocery inventory manager, a digital recipe and meal planner, and a household chore tracker. The system distinguishes itself through a REST API that exposes all household functions for home automation integration and a barcode-based stock controller that utilizes external databases to identify products and update stock levels. It also features recipe web scraping to automatically extract culinary data from external websi

RSS-Bridge is a self-hosted feed generator and proxy that transforms website content from sources without native feeds into standardized Atom, RSS, or JSON web feeds. It functions as a web scraping system that extracts data from pages using CSS selectors and XPath to create structured data streams for feed readers. The project is designed for extensibility, allowing for the development of custom bridges to fetch and parse data from new target websites. It includes capabilities for feed aggregation and filtering, enabling the merging of multiple data sources into a single feed and the removal

Pup is a command line tool for extracting and filtering data from HTML documents using CSS selectors. It functions as a parser and selector engine that isolates specific elements based on tags, IDs, classes, and attributes. The project provides utilities for converting selected HTML nodes into plain text, attribute values, or structured JSON objects. It includes a markup formatter that corrects missing tags and applies consistent indentation to improve the readability of HTML documents. The tool handles the retrieval of text content and attributes through a CSS selector engine, supporting co

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

AutoMergePublicNodes is a system for scraping public proxy nodes from across the internet and merging them into a unified list. It serves as an aggregator and converter that transforms these collected nodes into specialized subscription formats for various network routing clients. The project includes dedicated generators for creating configuration files and subscriptions compatible with Clash, Clash Meta, Sing-Box, and V2Ray. It handles the conversion of aggregated lists into Base64 and other standardized formats required by these specific network applications. The software also provides a

Venera is a multi-source content reader and aggregator that allows users to browse and download media from various remote websites and local files through a unified interface. It functions as a local-remote media manager, synchronizing online content with local storage to enable offline viewing. The project utilizes a JavaScript-based content parser and aggregator to scrape and parse data from external web sources. This system allows for the definition of custom data extraction rules using JavaScript to fetch and display content from external websites. The platform covers remote media manage

youtube-transcript-api is a Python library designed to retrieve and download subtitles and captions from YouTube videos using video IDs. It functions as an API client that extracts text and timing data for video content. The project includes a wrapper for automated translation, allowing transcripts to be converted into different target languages. It also features a retrieval system that supports routing requests through HTTP, HTTPS, or SOCKS proxies to avoid IP blocking and regional restrictions. The library provides tools for identifying available subtitle tracks and converting raw transcri

PROXY-List este un agregator public de proxy-uri care oferă structuri de date pentru stocarea și agregarea adreselor de servere proxy HTTP și SOCKS disponibile public. Servește drept sursă pentru recuperarea listelor de rutare a traficului de rețea, utilizate pentru a masca adresele IP de origine în timpul cererilor web. Proiectul utilizează un pipeline de date pentru a scana, interoga și serializa automat listele de proxy de pe mai multe site-uri web publice. Această infrastructură asigură disponibilitatea serverelor active prin interogări periodice programate și reîmprospătări automate ale conținutului, livrând listele rezultate sub formă de fișiere text simple. Aceste capabilități susțin anonimizarea traficului de rețea și menținerea infrastructurii de web scraping prin distribuirea cererilor de ieșire într-un pool de adrese proxy publice rotative, pentru a evita limitările de rată și ban-urile IP.