dataabc/weibo-crawler
Weibo Crawler
这是一个新浪微博网页爬虫和社交媒体数据管道，旨在提取用户资料、帖子、评论和多媒体资源。它作为一个容器化的数据爬虫，自动化收集社交媒体内容和互动指标，并将其存储在本地。
该系统包含一个处理层，利用大语言模型分析抓取的文本，生成摘要和情感分析。它通过一个部署就绪的容器模型脱颖而出，该模型具有用于管理提取任务和监控作业进度的 HTTP 界面。
该爬虫涵盖了广泛的功能，包括通过定时增量更新进行社交媒体监控、将多媒体资源归档到本地磁盘，以及向平面文件或数据库进行多格式数据导出。它还能捕获详细的社交互动，如一级评论和转发。
Features
- Social Media Platform Extraction - Specializes in the extraction of structured text and multimedia data from the Weibo social platform.
- Social Media Post Retrievers - Provides a system for retrieving collections of captured social media posts or single posts via unique identifiers.
- User Profile Post Extraction - Provides targeted retrieval of user profiles and their associated posts including engagement metrics.
- Social Platform Data Extraction - Retrieves detailed post information including timestamps, interaction counts, hashtags, and publication tools.
- Media File Storages - Downloads and saves original images and videos from social media posts directly to the local filesystem.
- Multi-Format Data Persistence - Supports persisting extracted content across flat files, relational databases, and document databases.
- Social Interaction Mapping - Captures and structures social interaction data, such as first-level comments and reposts, associated with posts.
- Social Media Data Pipelines - Automates the collection, incremental updating, and local storage of social media content and metrics.
- Container Deployment - Packages the crawler and its dependencies into a container for consistent deployment and automated restarts.
- Social Media Asset Downloaders - Provides capabilities to download original images, videos, and live photos from captured social media posts.
- Containerized - Provides a deployment-ready containerized crawler with an HTTP interface for task management.
- Background Task Schedulers - Implements automated background processes to run periodic data refresh tasks on a fixed timetable.
- Data Collection Schedulers - Uses schedulers to trigger the fetching of social media data at fixed time intervals.
- Incremental Crawling - Tracks the date of the last successful run to fetch only new posts and avoid duplicates.
- Social Media Monitoring - Tracks new posts and user activity over time using scheduled incremental crawls.
- User Profile Extractions - Collects comprehensive user profile details including nicknames, registration dates, and verification status.
- Web Scrapers - Extracts structured user profiles, posts, and multimedia assets from a social media platform.
- LLM-Based Text Processing - Utilizes large language models to perform text transformations such as summarization and sentiment analysis.
- Social Media Sentiment Analysis - Uses large language models to perform sentiment analysis and generate summaries from short-form social media posts.
- Containerized Deployments - Packages the crawler and its dependencies into containers for consistent execution and automated recovery.
- Crawl Initiators - Provides an HTTP interface for programmatically triggering and tracking the progress of web crawling jobs.
- Data Refreshers - Implements automated mechanisms for updating local datasets with fresh content from social media platforms.
- Management APIs - Offers HTTP interfaces to manage extraction tasks, monitor system status, and trigger data updates.
- HTTP Control Interfaces - Ships a RESTful interface used to trigger crawling jobs and monitor their execution progress.
