Open-source alternatives to WeiboSpider

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

WeiboSpider is a social media scraper designed to extract user profiles, posts, and interaction data from the Sina Weibo platform. It functions as a web-based data crawler that retrieves information via external interfaces rather than parsing the visual frontend. The tool includes a content lineage tracer to follow shared posts back to their original sources. It also features a social engagement analyzer to collect view counts and nested comment threads to measure user interaction metrics. The system provides capabilities for keyword-based social monitoring and search result filtering to tra

snscrape is a Python-based social media web scraper and crawler designed to extract public posts, profiles, and hashtags from social networks without the use of official APIs. It functions as an archival tool and a utility for open-source intelligence data collection, allowing for the gathering of publicly available information to investigate trends and people. The tool facilitates social media data extraction for research and archival purposes, enabling the creation of historical records of conversations and user activity. It supports workflows for academic social analysis and the export of

JMComic-Crawler-Python is a high-performance asynchronous web scraper and API client designed to programmatically retrieve images and metadata from a comic hosting service. It functions as a media archiving tool for batch downloading albums and chapters, automating the process of saving content to a local filesystem. The project is distinguished by its ability to reverse server-side pixel obfuscation, using a decryption tool to reconstruct sliced and shuffled images. To maintain stable connectivity, it utilizes a network bypass utility featuring dynamic domain rotation and proxy routing to ci

dlt is a Python data ingestion tool and ETL pipeline framework designed to fetch data from diverse sources and persist it into structured destinations. It functions as a schema inference engine that automatically detects data types and flattens nested JSON structures into relational tables, moving data from sources to lakehouses, warehouses, or vector databases. The project distinguishes itself through AI-powered pipeline generation, using large language models to scaffold extraction code and connectors for REST APIs. It also supports multimodal vector storage and specialized population of ve

Blynk is an embedded device framework and IoT cloud connectivity library designed to establish secure, bi-directional communication between microcontrollers and a remote management platform. It provides the core identity of an IoT device management tool, enabling the synchronization of device states, remote hardware control, and the mapping of hardware data to cloud-based interfaces. The project distinguishes itself through a virtual-pin system that decouples cloud communication from physical pins, allowing for hardware-independent data exchange. It also supports advanced architectural offloa

FxEmbed is a collection of specialized services providing a social media data API, a social media embed gateway, and a URL unshortener and sanitizer. It functions as an edge-deployed content proxy designed to programmatically fetch posts, threads, profiles, and search results from various social platforms. The project transforms social media links into rich media previews and interactive embeds for messaging platforms. It also expands shortened links to their original destinations while removing tracking parameters to improve user privacy and security. The system includes capabilities for so

This is a tool for searching, downloading, and archiving articles and engagement metadata from WeChat official accounts. It functions as a web-based content scraper and data exporter, allowing for the automated retrieval of social media content and the collection of performance metrics. The project distinguishes itself through a system that captures session credentials and authentication cookies from desktop clients via a local proxy to access private engagement data. It utilizes a concurrent proxy-pool fetching mechanism to download large volumes of content while avoiding rate limits, and it

MediaCrawler is an automated web scraping framework designed to extract public posts, comments, and creator metadata from various social media platforms. It functions as a headless browser automator, utilizing real browser instances to render dynamic content and execute the client-side scripts necessary for interacting with modern web interfaces. The system distinguishes itself through a focus on session persistence and network flexibility. It supports remote debugging to reuse active browser sessions and cookies, which helps minimize the risk of triggering platform security challenges. To ma

VidBee is a self-hosted media download manager that wraps the yt-dlp engine to download videos and audio from over 1000 websites. It functions as both a desktop client and a Fastify-based web service, managing downloads through a persistent queue with pause, resume, retry, and real-time progress tracking. The application uses cookie-based authentication to access login-gated, age-restricted, or subscriber-only content by importing browser cookies or Netscape-format cookie files. The application distinguishes itself through automated download workflows, including RSS and Atom feed monitoring t

Han1meViewer is an Android media viewer application for browsing, streaming, and downloading media content from a specific external website. It functions as a privacy-focused media browser that adapts external site content to a mobile-optimized interface. The application features tools for bypassing network restrictions through proxy and CDN configuration. It provides privacy protections including application locks and launcher icon disguises to hide the application's purpose. The project covers a wide range of capabilities, including background video downloading for offline media management

MechanicalSoup is a Python web automation library and scraping framework designed to simulate browser sessions and navigate websites without requiring JavaScript execution. It functions as an HTML parsing tool and HTTP session manager, allowing for the programmatic retrieval of page content and the automation of web interactions. The library distinguishes itself by combining session persistence with automated form interaction. It maps user data to HTML input fields and selection boxes for programmatic submission and maintains authenticated states by managing cookies and user-agent headers acr

InfoSpider is a personal data aggregator and digital footprint analyzer. It extracts user activity and history from social platforms and local browser database files to consolidate information into a unified format. The system functions as a social media archiving tool that converts feed data and albums from external links into downloadable PDF documents for offline preservation. It also serves as a browser history extractor that reads local SQLite database files to retrieve and analyze web navigation history. The project covers capabilities for data aggregation, digital footprint analysis,

This project is a promise-based HTTP client for Node.js that wraps callback-style network requests into promise objects. It serves as a library for sending JSON, form data, and multipart requests to web servers while managing asynchronous flow control. The library utilizes a specialized Bluebird promise implementation to handle request and response lifecycles. It includes a session manager that uses a cookie jar to maintain and transmit state across multiple network calls. The tool covers a broad range of network capabilities, including REST API integration, response data transformation, and

Revel is a full-stack web framework and toolkit for building applications with the Go language. It implements a model-view-controller architecture to separate business logic from user interface rendering, providing a comprehensive system for routing, parameter binding, and session management. The project distinguishes itself with a high-productivity development environment featuring automatic code compilation and hot-reloading, which refreshes the application state and templates upon file changes without requiring manual restarts. It also employs reflection-based parameter binding to automati

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect, process, and route structured information from websites. It functions as a comprehensive pipeline that manages the entire lifecycle of data gathering, from initial configuration to final storage in external databases or message queues. The platform distinguishes itself through a visual configuration interface that allows users to define extraction rules and manage scraping templates without writing custom code. It supports both static and dynamic content retrieval by integratin

scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields. The tool uses CSS selectors to target specific data points and employs schema-driven data mapping to organize unstructured web text into a consistent format. It supports custom value transformation to convert raw extracted strings into specific data formats. The system provides capabilities for web data extraction and automated cont

WechatSogou is a Python library that provides a programmatic bridge to WeChat content by scraping Sogou's search engine results. It resolves WeChat account profile pages to article lists, filters trending articles by predefined categories, and manages browser-like cookie sessions to maintain authenticated access. The library parses raw HTML to extract article metadata, account details, and keyword suggestions, while dynamically adjusting request frequency to avoid IP blocking. The project offers capabilities for searching WeChat public accounts and articles by keyword, returning profile detai

This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It functions as a social media data extractor that simulates browser requests to gather information without the need for official API keys or user account authentication. The tool provides capabilities for gathering public posts, harvesting user profile metadata such as biographies and locations, and retrieving trending topics categorized by geographical region. It can perform targeted content scraping based on specific usernames, hashtags, or search queries. The system manages data

TodoApp is a task management web application designed for organizing and tracking pending items. It consists of a web-based interface and a REST API backend that handles business logic and data requests. The system includes an OAuth 2.0 identity provider for user authentication via passwords and external social providers, as well as an API gateway proxy that routes traffic from the frontend to the backend to prevent cross-origin resource sharing issues. Operational capabilities cover system observability through OpenTelemetry for collecting logs and metrics, request rate limiting to maintain

WeChatVideoDownloader is a social media video extractor designed to retrieve and save video content from WeChat Channels for offline local storage. It functions as a content archiver that captures multimedia posts from the WeChat ecosystem to prevent data loss. The tool enables the downloading of video files from social channels, allowing for social media archiving and offline content consumption within different media players.

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

Telegram-Media-Downloader is a web media scraper and restricted content downloader designed to extract images and videos from the Telegram web interface for local storage. It functions by programmatically automating a web browser to capture media from chats and stories. The utility specifically enables the recovery of restricted content from groups and channels where administrators have disabled the native download function. It bypasses these restrictions by identifying media elements within the browser to locate source URLs. The system includes capabilities for session-based authentication,

Komga is a self-hosted digital comics and eBook server that organizes, serves, and streams CBZ, CBR, PDF, and EPUB files through a web interface. Built as a Java Spring Boot application with SQLite-based metadata storage, it provides multi-user access controls with individual reading progress tracking and supports the OPDS protocol for compatibility with third-party reader applications. The server distinguishes itself through comprehensive authentication options, including OAuth2 and OpenID Connect support with automatic account creation and email verification, alongside traditional cookie an