Open-source alternatives to WeChatVideoDownloader

InfoSpider is a personal data aggregator and digital footprint analyzer. It extracts user activity and history from social platforms and local browser database files to consolidate information into a unified format. The system functions as a social media archiving tool that converts feed data and albums from external links into downloadable PDF documents for offline preservation. It also serves as a browser history extractor that reads local SQLite database files to retrieve and analyze web navigation history. The project covers capabilities for data aggregation, digital footprint analysis,

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

wechatVideoDownload is a tool for capturing, downloading, and decrypting video content from the WeChat application for local storage. It functions as a batch media downloader, a channel content scraper, and a media decryption utility. The project distinguishes itself by providing specific capabilities for WeChat video archiving, including the processing of encrypted files into standard playable formats and the extraction of content from channels and live stream replays. The system employs packet-sniffing traffic analysis and pattern-based URL extraction to identify media links during active

so-novel is a web novel downloader and scraping engine designed to extract structured text from websites and convert it into electronic book formats. It functions as a multi-interface content extractor, providing a shared backend accessible via a web-based management dashboard, a terminal user interface, and a command line interface. The system utilizes a rule-driven approach for data extraction, using CSS selectors and XPath rules defined in external configuration files to map web elements to specific data fields. To maintain access to content, it includes a proxy-routed request pipeline to

Playwright for Python is a browser automation framework designed for end-to-end testing, web scraping, and user interaction simulation. It functions as a headless browser controller that enables programmatic navigation, data extraction, and the execution of complex workflows across multiple rendering engines. The framework distinguishes itself through an actionability-aware interaction engine that automatically verifies element readiness before performing actions, significantly reducing test flakiness. It utilizes isolated browser contexts to maintain separate storage and cookies for parallel

Firecrawl MCP Server is a Model Context Protocol tool server that exposes the full suite of Firecrawl’s web scraping, crawling, and automation capabilities as tools that large language models can invoke directly. It acts as a proxy to the Firecrawl cloud platform, which manages headless browser orchestration, async job queues, and rate limiting behind the scenes. The server distinguishes itself by packaging autonomous web agents — both a research agent that browses and collects structured data from multiple pages, and a general web agent that performs multi-step browsing and extraction tasks

FreeTube is a privacy-focused desktop application for watching YouTube videos without ads, tracking cookies, or the requirement of a Google account. It functions as a local-first subscription manager that tracks channels and playlists in local files instead of a centralized cloud account. The application avoids tracking-heavy official APIs by using a content extractor that parses web pages directly. To further protect user identity, it can route network traffic through proxies or Tor to mask the hardware IP address. The software provides tools for distraction-free viewing, including the abil

This project is a community discussion forum and management suite for developers. It provides a space for technical questioning, resource sharing, and project showcasing, supported by a contributor reward system that uses leaderboards to track and recognize helpful community members. The system includes a migration tool specifically for importing and archiving community conversations from social platforms into GitHub Discussions. It also features a notification bridge that synchronizes forum activity with external chat channels to provide real-time alerts for new posts. The platform supports

howdoi is a command-line coding answer engine that retrieves programming solutions and code snippets from the web for display directly in the terminal. It functions as a web-based code search tool that uses natural language queries to find technical answers without requiring a web browser. The tool provides a JSON-exportable query system, allowing search results to be output as structured data for integration with other software and text editors. It features terminal-based knowledge retrieval that includes local caching and stashing of answers to reduce network latency and avoid search engine

MusicDownload is a command-line tool for downloading songs from the QQ Music platform, supporting both lossless FLAC and compressed MP3 audio formats. It wraps QQ Music's internal API endpoints to search for tracks, retrieve metadata, and obtain download links. The tool operates through a text-based interface where users input search queries and trigger downloads, with configuration file parsing allowing customization of download paths, format preferences, and API keys. Error handling middleware manages API failures and network errors to ensure graceful degradation when issues occur. Audio f

This project is a Python-based web scraping tool and command line image downloader designed to automate the retrieval of images from Google Images. It functions as an image dataset collector, allowing users to gather large sets of images for data analysis or research through a terminal interface or programmatic scripts. The tool features advanced search filtering to restrict results by file format, color, size, aspect ratio, and usage rights. It also supports reverse image search to find visually similar media based on a provided URL and offers search scope expansion to increase result volume

Movie Data Capture is a media library organizer and movie metadata scraper designed to automatically categorize and name files in a local media collection. It functions as an automated content tagger that identifies movie files and applies descriptive tags by extracting film details from web databases. The system utilizes an HTTP web scraper to fetch information from external APIs and remote HTML content. It employs a filename pattern parser to extract movie titles and release years from local files using regular expressions, which are then used to automate search queries. The tool maps scra

This project is a collection of Python implementations for web scraping, network traffic interception, data analysis, and sentiment analysis. It provides methods for extracting structured data from websites and mobile application interfaces. The collection includes tools for capturing and analyzing network packets from mobile applications to identify hidden internal API endpoints. It also features scripts for evaluating the emotional tone and public perception of text data. The project covers data manipulation and transformation of large datasets, as well as the generation of charts and grap

Robin is an AI-powered open source intelligence framework and dark web investigation tool. It functions as a multi-model AI orchestrator that integrates search engines and web scrapers with language models to automate information gathering and data synthesis. The system utilizes a crawl-and-filter architecture to isolate high-value data from raw web content and employs a query-refinement pipeline to optimize search terms. It specifically supports dark web investigations by routing requests through proxies to access hidden services and using language models to analyze and summarize findings fr

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

This is a build tool for adding progressive web app capabilities to Vite projects. It automates the generation of web app manifests and service workers, enabling websites to become installable on mobile and desktop devices. The project provides a specialized asset pipeline that generates a complete set of required application icons and theme colors from a single source image. It also manages the service worker lifecycle, offering both automatic generation and the ability to compile custom service worker files with an integrated precache manifest. The plugin covers broad capability areas incl

Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo social network. It functions as a social media data extractor that utilizes a distributed task queue to scale scraping operations across multiple worker nodes. The system includes a graphical administrative interface for configuring crawler settings, target user identifiers, and search keywords. It employs a distributed architecture to increase data throughput and manage large-scale collection of social media content. The tool covers a wide range of data collection capabilities,

next-pwa is a plugin for adding progressive web app capabilities and offline support to Next.js applications. It provides a configuration layer and a Workbox-based generator to automate the creation, registration, and installation of service workers and web app manifests. The tool enables developers to implement custom service worker logic by injecting TypeScript or JavaScript scripts to define specialized caching strategies and request interception behaviors. It also includes mechanisms for detecting application updates and prompting users to refresh the page when a new version is available.

snscrape is a Python-based social media web scraper and crawler designed to extract public posts, profiles, and hashtags from social networks without the use of official APIs. It functions as an archival tool and a utility for open-source intelligence data collection, allowing for the gathering of publicly available information to investigate trends and people. The tool facilitates social media data extraction for research and archival purposes, enabling the creation of historical records of conversations and user activity. It supports workflows for academic social analysis and the export of

Jsoup is a Java library designed for parsing, extracting, and manipulating HTML and XML content. It provides a document object model that represents web content as a hierarchical tree, allowing for programmatic navigation and modification of elements, attributes, and text. The library functions as a toolkit for web scraping, enabling the retrieval of remote content via standard web protocols and the management of HTTP sessions for automated form interaction. The library distinguishes itself through its fault-tolerant tokenization, which reconstructs valid document structures from malformed or

Turndown is a JavaScript library designed to transform HTML documents into structured Markdown. It functions as a flexible engine that parses web content by traversing the document object model and applying rule-based transformations to convert elements into their corresponding text-based syntax. The tool distinguishes itself through a modular architecture that allows for extensive customization of the conversion process. Users can define custom conversion rules to handle specific elements, implement content filtering to discard unwanted nodes, and configure character escaping to ensure outpu

Yao is an LLM agent framework and low-code web app builder designed for orchestrating autonomous AI agents. It provides a platform to design, deploy, and coordinate agents with specialized personas that can plan tasks, utilize external tools, and execute multi-stage pipelines. The project distinguishes itself through a Model Context Protocol server for connecting assistants to external binaries and HTTP services, and a gRPC remote execution engine that allows agents to manage remote servers and devices. It includes a model-agnostic provider bridge that supports dynamic switching between vario

node-lessons is a comprehensive Node.js programming course and instructional guide. It provides a collection of guided lessons and code examples designed to teach the fundamentals of the Node.js runtime and server-side JavaScript development. The project serves as a practical guide for building web servers and backend applications, specifically covering the implementation of HTTP servers, request routing, and middleware chains. It includes specialized instructional material on managing asynchronous JavaScript workflows through promises and flow control, as well as guides for integrating NoSQL

Claude-engineer is an autonomous software engineering agent and command-line interface for interacting with the Claude 3.5 Sonnet model. It functions as an AI code editor that writes code, manages local files, and executes terminal commands to automate technical workflows. The system features a self-evolving tool framework that allows the agent to design and implement its own functional scripts to expand its capabilities during a session. It utilizes a sandboxed Python executor to run scripts for data analysis and complex computations in a secure remote environment. The project covers a broa

Grequests is an asynchronous HTTP batcher and Gevent-based client library used to execute large sets of network requests simultaneously. It functions as a concurrent request wrapper for the Requests library, enabling non-blocking operations to reduce the total time spent waiting for server responses. The project provides a task-pool execution model to handle batch network operations, such as high-throughput web scraping and API polling. It can stream responses as they arrive via a generator, allowing for immediate data processing without waiting for the entire batch to complete. The library

Open Deep Research is an AI-powered web research agent that combines a reasoning model with live web search and data extraction to perform deep, multi-source investigations on any topic. It operates through a dual interface, offering both a command-line tool and a Model Context Protocol server, allowing developers to integrate web capabilities directly into AI agents and coding assistants. The project distinguishes itself by orchestrating an iterative research loop where a reasoning model plans steps, interprets search results, and guides subsequent web interactions. It uses Firecrawl for scr