awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
dataabc avatar

dataabc/weibo-crawler

0
View on GitHub↗

Weibo Crawler

This project is a Sina Weibo web scraper and social media data pipeline designed to extract user profiles, posts, comments, and multimedia assets. It functions as a containerized data crawler that automates the collection and local storage of social media content and engagement metrics.

The system includes a processing layer that uses large language models to analyze scraped text, generating summaries and sentiment analysis. It differentiates itself through a deployment-ready container model featuring an HTTP interface for managing extraction tasks and monitoring job progress.

The crawler covers a broad range of capabilities, including social media monitoring via scheduled incremental updates, multimedia asset archiving to local disks, and multi-format data export to flat files or databases. It also captures detailed social interactions, such as first-level comments and reposts.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Social Media Platform Extraction - Specializes in the extraction of structured text and multimedia data from the Weibo social platform.
  • Social Media Post Retrievers - Provides a system for retrieving collections of captured social media posts or single posts via unique identifiers.
  • User Profile Post Extraction - Provides targeted retrieval of user profiles and their associated posts including engagement metrics.
  • Social Platform Data Extraction - Retrieves detailed post information including timestamps, interaction counts, hashtags, and publication tools.
  • Media File Storages - Downloads and saves original images and videos from social media posts directly to the local filesystem.
  • Multi-Format Data Persistence - Supports persisting extracted content across flat files, relational databases, and document databases.
  • Social Interaction Mapping - Captures and structures social interaction data, such as first-level comments and reposts, associated with posts.
  • Social Media Data Pipelines - Automates the collection, incremental updating, and local storage of social media content and metrics.
  • Container Deployment - Packages the crawler and its dependencies into a container for consistent deployment and automated restarts.
  • Social Media Asset Downloaders - Provides capabilities to download original images, videos, and live photos from captured social media posts.
  • Containerized - Provides a deployment-ready containerized crawler with an HTTP interface for task management.
  • Background Task Schedulers - Implements automated background processes to run periodic data refresh tasks on a fixed timetable.
  • Data Collection Schedulers - Uses schedulers to trigger the fetching of social media data at fixed time intervals.
  • Incremental Crawling - Tracks the date of the last successful run to fetch only new posts and avoid duplicates.
  • Social Media Monitoring - Tracks new posts and user activity over time using scheduled incremental crawls.
  • User Profile Extractions - Collects comprehensive user profile details including nicknames, registration dates, and verification status.
  • Web Scrapers - Extracts structured user profiles, posts, and multimedia assets from a social media platform.
  • LLM-Based Text Processing - Utilizes large language models to perform text transformations such as summarization and sentiment analysis.
  • Social Media Sentiment Analysis - Uses large language models to perform sentiment analysis and generate summaries from short-form social media posts.
  • Containerized Deployments - Packages the crawler and its dependencies into containers for consistent execution and automated recovery.
  • Crawl Initiators - Provides an HTTP interface for programmatically triggering and tracking the progress of web crawling jobs.
  • Data Refreshers - Implements automated mechanisms for updating local datasets with fresh content from social media platforms.
  • Management APIs - Offers HTTP interfaces to manage extraction tasks, monitor system status, and trigger data updates.
  • HTTP Control Interfaces - Ships a RESTful interface used to trigger crawling jobs and monitor their execution progress.
4,541 stars·903 forks·Python·83 views

Star history

Star history chart for dataabc/weibo-crawlerStar history chart for dataabc/weibo-crawler

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does dataabc/weibo-crawler do?

This project is a Sina Weibo web scraper and social media data pipeline designed to extract user profiles, posts, comments, and multimedia assets. It functions as a containerized data crawler that automates the collection and local storage of social media content and engagement metrics.

What are the main features of dataabc/weibo-crawler?

The main features of dataabc/weibo-crawler are: Social Media Platform Extraction, Social Media Post Retrievers, User Profile Post Extraction, Social Platform Data Extraction, Media File Storages, Multi-Format Data Persistence, Social Interaction Mapping, Social Media Data Pipelines.

Which projects share features with dataabc/weibo-crawler?

Projects with overlapping indexed features include: dataabc/weibospider — weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement… spiderclub/weibospider — Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo… bisguzar/twitter-scraper — This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It… nghuyong/weibospider — WeiboSpider is a social media scraper designed to extract user profiles, posts, and interaction data from the Sina… sdelements/lets-chat — Lets-chat is a self-hosted team communication platform and XMPP chat server designed for private messaging. It… subzeroid/instagrapi.

Projects sharing features with Weibo Crawler

These projects share indexed features with Weibo Crawler. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • dataabc/weibospiderdataabc avatar

    dataabc/weiboSpider

    9,630View on GitHub↗

    weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

    Pythonhelp-wantedpythonpython3
    View on GitHub↗9,630
  • spiderclub/weibospiderSpiderClub avatar

    SpiderClub/weibospider

    4,787View on GitHub↗

    Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo social network. It functions as a social media data extractor that utilizes a distributed task queue to scale scraping operations across multiple worker nodes. The system includes a graphical administrative interface for configuring crawler settings, target user identifiers, and search keywords. It employs a distributed architecture to increase data throughput and manage large-scale collection of social media content. The tool covers a wide range of data collection capabilities,

    Pythondata-analysisdistributed-crawlerpython3
    View on GitHub↗4,787
  • bisguzar/twitter-scraperbisguzar avatar

    bisguzar/twitter-scraper

    4,013View on GitHub↗

    This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It functions as a social media data extractor that simulates browser requests to gather information without the need for official API keys or user account authentication. The tool provides capabilities for gathering public posts, harvesting user profile metadata such as biographies and locations, and retrieving trending topics categorized by geographical region. It can perform targeted content scraping based on specific usernames, hashtags, or search queries. The system manages data

    Pythonclientjavascriptno-authentication
    View on GitHub↗4,013
  • nghuyong/weibospidernghuyong avatar

    nghuyong/WeiboSpider

    4,086View on GitHub↗

    WeiboSpider is a social media scraper designed to extract user profiles, posts, and interaction data from the Sina Weibo platform. It functions as a web-based data crawler that retrieves information via external interfaces rather than parsing the visual frontend. The tool includes a content lineage tracer to follow shared posts back to their original sources. It also features a social engagement analyzer to collect view counts and nested comment threads to measure user interaction metrics. The system provides capabilities for keyword-based social monitoring and search result filtering to tra

    Pythonpythonscrapyweibo
    View on GitHub↗4,086
Compare all 30 related projects→