dataabc/weiboSpider
WeiboSpider
weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content.
The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamps.
Capabilities include keyword-based post searches within defined time windows, the harvesting of original images and videos, and social network mapping via the extraction of follower lists. Extracted data can be filtered by date or originality and is persisted to flat files, relational databases, document databases, or transmitted as token-authenticated JSON payloads to remote API endpoints.
Features
- Social Platform Data Extraction - Gathers detailed user profiles and published posts from Sina Weibo for academic research and trend analysis.
- Data Extraction - Implements a specialized system for extracting posts, profiles, and engagement metrics from Sina Weibo.
- Social Media Post Retrievers - Collects posts and engagement metrics from social networks for academic research and analysis.
- Social Media Archiving Tools - Schedules regular crawls to permanently save and organize content from social media platforms.
- Data Extraction Pipelines - Automates the workflow of parsing and synchronizing social media data into structured JSON payloads.
- Incremental Data Collection - Resumes extraction from the last successful crawl timestamp to capture only new posts.
- Social Profile Extractors - Gathers long-form text, images, and videos from Sina Weibo profiles using session cookies.
- Session & Cookie Handlers - Retrieves session cookies from browser developer tools to maintain authenticated sessions for data extraction.
- Cookie-Based Authentication Bridges - Utilizes browser session cookies to authenticate requests and access protected user profiles on Sina Weibo.
- Incremental Crawling - Tracks the most recent post date per user to capture only new content since the last execution.
- Client-Side Request Pacing - Implements randomized wait intervals between requests to mimic human behavior and avoid platform rate limits.
- Web Scrapers - Functions as a Python-based tool for extracting structured data from the Sina Weibo platform.
- Keyword Search Crawlers - Queries the platform search API by keyword to collect matching posts within specific time ranges.
- User Identifier Harvesting - Extracts specific user IDs from profile pages or discovers new IDs by crawling following lists.
- Public Account Following Retrievers - Retrieves complete lists of accounts that a public social media user follows.
- Automated Extraction Schedulers - Runs automated extraction processes at regular intervals to capture new social media content.
- Full-Text Content Extraction - Extracts full-text versions of long-form posts by visiting the specific detail pages of the social platform.
- Temporal Range Filtering - Allows users to define specific start and end dates to collect posts within a designated time window.
- Keyword-Based Content Collectors - Retrieves social media posts containing specific keywords within defined time windows.
- Keyword-Based Trend Monitoring - Searches for and extracts posts containing specific terms within defined time windows for trend analysis.
- Multi-Destination Data Routing - Routes extracted social media data to multiple destinations including flat files, relational databases, and document databases.
- Multi-Format Data Persistence - Persists extracted information across various storage types, including flat files and relational or document databases.
- Date-Based Filters - Limits the extraction of posts to a specific time window using configurable date parameters.
- Social Network Discovery - Expands the target user list by recursively extracting identifiers from the following lists of crawled accounts.
- Social Media Content Extractors - Provides automated downloading of original images, videos, and live photos from social media posts.
- Social Relationship Maps - Extracts follower lists and user identifiers to visualize and analyze relationships between accounts.
- Social Media Asset Downloaders - Saves original images, videos, and live photo formats from posts and retweets to local directories.
- Community Content Details - Navigates to detailed post pages to retrieve complete long-form text when summary views are truncated.
