dataabc/weiboSpider

Features

• Social Platform Data Extraction - Gathers detailed user profiles and published posts from Sina Weibo for academic research and trend analysis.
• Data Extraction - Implements a specialized system for extracting posts, profiles, and engagement metrics from Sina Weibo.
• Social Media Post Retrievers - Collects posts and engagement metrics from social networks for academic research and analysis.
• Social Media Archiving Tools - Schedules regular crawls to permanently save and organize content from social media platforms.
• Data Extraction Pipelines - Automates the workflow of parsing and synchronizing social media data into structured JSON payloads.
• Incremental Data Collection - Resumes extraction from the last successful crawl timestamp to capture only new posts.
• Social Profile Extractors - Gathers long-form text, images, and videos from Sina Weibo profiles using session cookies.
• Session & Cookie Handlers - Retrieves session cookies from browser developer tools to maintain authenticated sessions for data extraction.
• Cookie-Based Authentication Bridges - Utilizes browser session cookies to authenticate requests and access protected user profiles on Sina Weibo.
• Incremental Crawling - Tracks the most recent post date per user to capture only new content since the last execution.
• Client-Side Request Pacing - Implements randomized wait intervals between requests to mimic human behavior and avoid platform rate limits.
• Web Scrapers - Functions as a Python-based tool for extracting structured data from the Sina Weibo platform.
• Keyword Search Crawlers - Queries the platform search API by keyword to collect matching posts within specific time ranges.
• User Identifier Harvesting - Extracts specific user IDs from profile pages or discovers new IDs by crawling following lists.
• Public Account Following Retrievers - Retrieves complete lists of accounts that a public social media user follows.
• Automated Extraction Schedulers - Runs automated extraction processes at regular intervals to capture new social media content.
• Full-Text Content Extraction - Extracts full-text versions of long-form posts by visiting the specific detail pages of the social platform.
• Temporal Range Filtering - Allows users to define specific start and end dates to collect posts within a designated time window.
• Keyword-Based Content Collectors - Retrieves social media posts containing specific keywords within defined time windows.
• Keyword-Based Trend Monitoring - Searches for and extracts posts containing specific terms within defined time windows for trend analysis.
• Multi-Destination Data Routing - Routes extracted social media data to multiple destinations including flat files, relational databases, and document databases.
• Multi-Format Data Persistence - Persists extracted information across various storage types, including flat files and relational or document databases.
• Date-Based Filters - Limits the extraction of posts to a specific time window using configurable date parameters.
• Social Network Discovery - Expands the target user list by recursively extracting identifiers from the following lists of crawled accounts.
• Social Media Content Extractors - Provides automated downloading of original images, videos, and live photos from social media posts.
• Social Relationship Maps - Extracts follower lists and user identifiers to visualize and analyze relationships between accounts.
• Social Media Asset Downloaders - Saves original images, videos, and live photo formats from posts and retweets to local directories.
• Community Content Details - Navigates to detailed post pages to retrieve complete long-form text when summary views are truncated.

Star history