Jack-Cherish/python-spider
Python Spider
This is a collection of Python scripts designed for extracting data from popular Chinese websites and mobile applications. It functions as a multi-platform data extraction toolkit, capable of automating tasks such as downloading videos from platforms like Bilibili and Douyin, scraping product reviews and images from e-commerce sites like Taobao and JD.com, and booking train tickets on the 12306 railway system.
The project distinguishes itself through its focus on automating specific, high-value tasks within the Chinese internet ecosystem. It includes capabilities for solving Chinese CAPTCHA challenges like GEETEST, removing watermarks from downloaded videos, and building a pool of proxy IPs to avoid blocking during large-scale scraping operations. A notable feature is its ability to assist with live quiz games by capturing questions from streaming apps, searching for answers online, and broadcasting the results in real time via WebSocket.
Beyond these differentiators, the toolkit covers a broad range of standard web scraping techniques. It handles both static and dynamic web content, manages session-based authentication for sites like Taobao, and provides utilities for downloading various media types including images, music, and novels. The project also includes scripts for querying university academic systems and scheduling automated actions, such as booking tickets at a precise time.
Features
- Web Scraping - A collection of Python scripts for scraping data from popular Chinese websites and apps.
- Live Quiz Answerers - Captures live quiz questions from streaming apps and automatically searches for and broadcasts answers in real time.
- Automated Ticket Booking Systems - Submits purchase requests to a ticketing system at a scheduled time to secure a train seat.
- Chinese E-Commerce Scrapers - Scrapes product listings, reviews, and images from major Chinese e-commerce platforms like Taobao and JD.com.
- Multi-Platform Media Downloads - Downloads multiple videos, images, or music files from platforms like Bilibili, JD, and Netease.
- Railway Ticket Booking Automators - Submits purchase requests to the 12306 railway booking system at scheduled times to secure train tickets.
- Railway Booking Automation - Submits purchase requests to the 12306 railway booking system to secure tickets as soon as they become available.
- Multi-Platform Content Extractors - Extracts content from e-commerce, social media, video, and ticketing platforms using automated requests.
- Bilibili Video Downloads - Downloads videos and real-time comments from Bilibili using advanced techniques to bypass anti-scraping measures.
- Chinese Video Platform Scrapers - Downloads videos and comments from Chinese platforms like Bilibili and Douyin, including watermark removal.
- Quiz Facilitators - Fetches quiz questions via packet capture, searches for answers using Baidu Zhidao, and pushes results to a client.
- Scraping Platform Adapters - Provides a unified interface for downloading media and content from multiple Chinese platforms.
- HTML and JSON Parsers - Chains HTTP requests with HTML and JSON parsing to extract structured data from websites.
- API Reverse Engineering - Intercepts mobile app and web traffic to reverse-engineer undocumented API endpoints for data extraction.
- Quiz OCR Extractors - Extracts text from images using OCR to automate answering quiz questions on live-streaming platforms.
- Web Search Answer Retrievers - Automates searching Baidu Zhidao for quiz answers by submitting extracted questions and parsing results.
- CAPTCHA Solving - Automates the recognition and solving of GEETEST-style CAPTCHA challenges for scraping workflows.
- Chinese CAPTCHA Solvers - Automates the recognition and solving of Chinese GEETEST-style CAPTCHA challenges for scraping workflows.
- Web Content Scraping - Downloads images or videos from pages that load content asynchronously.
- Task Schedulers - Triggers automated actions at precise times using system timers or sleep loops for time-sensitive services.
- Video Downloaders - Captures video files from mobile apps by intercepting network requests.
- Douyin Video Downloads - Downloads videos from a Douyin user's profile by searching for the user and fetching each video's download URL.
- Watermark-Free Media Retrieval - Retrieves a video from a sharing URL and strips the platform's watermark using a third-party or direct API.
- Live Quiz Answer Broadcasters - Ships a WebSocket broadcaster that pushes live quiz answers to browser clients in real time.
- Proxy Pool Builders - Collects and manages a rotating pool of proxy IPs from public sources to avoid blocking during scraping.
- Proxy Pool Automation - Collects and manages a pool of proxy IPs from public sources to rotate during large-scale scraping operations.
- Proxy and Fingerprint Rotation - Distributes requests across a dynamic pool of proxy IPs to avoid rate limiting and IP-based blocking.
- Live Quiz Answer Assistants - Captures quiz questions from live-streaming apps and searches for answers in real time.
- Live Quiz Answer Automators - Captures live quiz questions from streaming apps and automatically searches for and broadcasts answers in real time.
- E-Commerce Login Simulators - Automates the login process on Taobao to enable authenticated scraping of user-specific data.
- OCR Captcha Solving - Automates GEETEST and other visual CAPTCHA solving using OCR and pattern-matching algorithms.
- Session-Cookie Persistences - Maintains login state across requests by storing and reusing session cookies from initial authentication.
- Watermark Removal - Uses browser automation to interact with a third-party service that strips watermarks from downloaded videos.
- Real-Time Data Pushing - Broadcasts scraped or computed results to connected clients instantly using persistent WebSocket connections.
