dataabc/weibo-crawler

Features

• Social Media Platform Extraction - Specializes in the extraction of structured text and multimedia data from the Weibo social platform.
• Social Media Post Retrievers - Provides a system for retrieving collections of captured social media posts or single posts via unique identifiers.
• User Profile Post Extraction - Provides targeted retrieval of user profiles and their associated posts including engagement metrics.
• Social Platform Data Extraction - Retrieves detailed post information including timestamps, interaction counts, hashtags, and publication tools.
• Media File Storages - Downloads and saves original images and videos from social media posts directly to the local filesystem.
• Multi-Format Data Persistence - Supports persisting extracted content across flat files, relational databases, and document databases.
• Social Interaction Mapping - Captures and structures social interaction data, such as first-level comments and reposts, associated with posts.
• Social Media Data Pipelines - Automates the collection, incremental updating, and local storage of social media content and metrics.
• Container Deployment - Packages the crawler and its dependencies into a container for consistent deployment and automated restarts.
• Social Media Asset Downloaders - Provides capabilities to download original images, videos, and live photos from captured social media posts.
• Containerized - Provides a deployment-ready containerized crawler with an HTTP interface for task management.
• Background Task Schedulers - Implements automated background processes to run periodic data refresh tasks on a fixed timetable.
• Data Collection Schedulers - Uses schedulers to trigger the fetching of social media data at fixed time intervals.
• Incremental Crawling - Tracks the date of the last successful run to fetch only new posts and avoid duplicates.
• Social Media Monitoring - Tracks new posts and user activity over time using scheduled incremental crawls.
• User Profile Extractions - Collects comprehensive user profile details including nicknames, registration dates, and verification status.
• Web Scrapers - Extracts structured user profiles, posts, and multimedia assets from a social media platform.
• LLM-Based Text Processing - Utilizes large language models to perform text transformations such as summarization and sentiment analysis.
• Social Media Sentiment Analysis - Uses large language models to perform sentiment analysis and generate summaries from short-form social media posts.
• Containerized Deployments - Packages the crawler and its dependencies into containers for consistent execution and automated recovery.
• Crawl Initiators - Provides an HTTP interface for programmatically triggering and tracking the progress of web crawling jobs.
• Data Refreshers - Implements automated mechanisms for updating local datasets with fresh content from social media platforms.
• Management APIs - Offers HTTP interfaces to manage extraction tasks, monitor system status, and trigger data updates.
• HTTP Control Interfaces - Ships a RESTful interface used to trigger crawling jobs and monitor their execution progress.

Star history