Projects sharing features with WechatSogou

Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo social network. It functions as a social media data extractor that utilizes a distributed task queue to scale scraping operations across multiple worker nodes. The system includes a graphical administrative interface for configuring crawler settings, target user identifiers, and search keywords. It employs a distributed architecture to increase data throughput and manage large-scale collection of social media content. The tool covers a wide range of data collection capabilities,

Hanko is an open-source identity provider and customer identity and access management system. It serves as a passkey authentication service and an OAuth and SAML SSO gateway, allowing applications to authenticate users and issue tokens via standard identity protocols. The project distinguishes itself through a strong focus on passwordless access using WebAuthn-based passkeys and email-based passcodes. It provides framework-agnostic authentication interfaces as customizable web components that can be embedded directly into web applications to handle login, registration, and profile management.

Devise is a comprehensive identity management system and authentication framework for Ruby on Rails applications. It provides a complete set of tools for managing user registration, secure sign-in, and session handling using a modular strategy pattern. The framework distinguishes itself by offering a suite of security hardening features, including brute force protection through account locking and secure password recovery workflows. It also functions as an integrator for external identity providers and third-party authentication via standardized protocols. Broad capabilities cover the full u

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically navigating websites and simulating human behavior without a graphical user interface. The library provides an automated interface for populating and submitting web forms, including text fields, checkboxes, and file uploads. It manages stateful sessions by automatically storing and sending cookies across multiple requests to maintain user authentication and identity. Additional capabilities include web data scraping, the ability to download remote web content, and the maintena

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

docetl is an AI-powered document ETL tool and map-reduce orchestrator designed to transform large collections of unstructured documents into structured, queryable tables using language models. It provides a declarative pipeline framework for extracting, cleaning, and transforming data from sources such as PDFs and text files into predefined schemas. The project distinguishes itself through a semantic data integration suite that enables joining datasets and resolving duplicate entities based on embedding-based similarity. It includes an interactive prompt playground for developing and optimizi

Supertest is a Node.js HTTP testing library that provides a framework for making requests and asserting responses from web servers. It functions as an HTTP assertion framework designed to verify status codes, response bodies, and headers during automated server tests. The library features an ephemeral port server manager that automatically binds servers to random available ports, removing the need for manual port tracking and preventing conflicts during concurrent test execution. It also includes an HTTP session state manager to persist cookies and session data across multiple sequential requ

iron-session is a stateless session management library for JavaScript that stores encrypted and signed session data directly in browser cookies. It functions as authentication middleware to verify user identity without requiring a server-side database or network lookups for session retrieval. The project provides a symmetric key encryption tool that supports versioned password rotation. This allows encryption keys to be updated across a system without invalidating active user sessions. Beyond session management, the library includes utilities to seal arbitrary data objects into secure signed

OpenPDF is a Java library and document processor used for creating, editing, rendering, and encrypting PDF documents. It functions as a toolkit for generating new files from scratch, modifying existing document structures, and extracting text content. The project includes a dedicated engine for transforming HTML and CSS content into PDF documents by parsing markup and applying styles. It also provides a rendering engine to convert PDF pages into image formats for thumbnails and previews, alongside a security utility for protecting content via document encryption. The library supports the add

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

TodoApp is a task management web application designed for organizing and tracking pending items. It consists of a web-based interface and a REST API backend that handles business logic and data requests. The system includes an OAuth 2.0 identity provider for user authentication via passwords and external social providers, as well as an API gateway proxy that routes traffic from the frontend to the backend to prevent cross-origin resource sharing issues. Operational capabilities cover system observability through OpenTelemetry for collecting logs and metrics, request rate limiting to maintain

pdf2htmlEX is a PDF to HTML converter that transforms documents into web pages while preserving the original layout, fonts, and formatting. It functions as a layout engine and text extractor, mapping PDF coordinate data to HTML and CSS to maintain visual fidelity. The tool converts PDF content into searchable and selectable native HTML text by embedding original document fonts. It maintains document interactivity by preserving internal links, bookmarks, and outlines, converting them into functional web navigation. The conversion process supports flexible output structures, allowing documents

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

Komga is a self-hosted digital comics and eBook server that organizes, serves, and streams CBZ, CBR, PDF, and EPUB files through a web interface. Built as a Java Spring Boot application with SQLite-based metadata storage, it provides multi-user access controls with individual reading progress tracking and supports the OPDS protocol for compatibility with third-party reader applications. The server distinguishes itself through comprehensive authentication options, including OAuth2 and OpenID Connect support with automatic account creation and email verification, alongside traditional cookie an

This project is a promise-based HTTP client for Node.js that wraps callback-style network requests into promise objects. It serves as a library for sending JSON, form data, and multipart requests to web servers while managing asynchronous flow control. The library utilizes a specialized Bluebird promise implementation to handle request and response lifecycles. It includes a session manager that uses a cookie jar to maintain and transmit state across multiple network calls. The tool covers a broad range of network capabilities, including REST API integration, response data transformation, and

Revel is a full-stack web framework and toolkit for building applications with the Go language. It implements a model-view-controller architecture to separate business logic from user interface rendering, providing a comprehensive system for routing, parameter binding, and session management. The project distinguishes itself with a high-productivity development environment featuring automatic code compilation and hot-reloading, which refreshes the application state and templates upon file changes without requiring manual restarts. It also employs reflection-based parameter binding to automati

Pup is a command line tool for extracting and filtering data from HTML documents using CSS selectors. It functions as a parser and selector engine that isolates specific elements based on tags, IDs, classes, and attributes. The project provides utilities for converting selected HTML nodes into plain text, attribute values, or structured JSON objects. It includes a markup formatter that corrects missing tags and applies consistent indentation to improve the readability of HTML documents. The tool handles the retrieval of text content and attributes through a CSS selector engine, supporting co

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

This project is a Python-based automation toolkit designed to manage programmatic authentication and session persistence across web services. It provides a framework for executing automated login sequences, including the handling of interactive security challenges such as QR code verification and captcha resolution. The toolkit distinguishes itself by simulating native mobile application environments, allowing for the execution of scripts that require specific device-level headers and behaviors. It also incorporates hook-based interception to monitor workflow states and manage exceptions duri

Pixez-flutter is a cross-platform mobile application designed for browsing, searching, and downloading digital artwork from external platforms. It provides a comprehensive interface for managing remote media collections, including support for viewing illustrations, manga, and animated content. The application handles user authentication and session persistence to ensure a consistent experience across different devices. The project distinguishes itself through a sophisticated network management layer that enables users to bypass regional restrictions and network limitations. By utilizing custo

This project is a request router and web framework for the Go programming language. It provides a toolkit for matching incoming HTTP network requests to specific handler functions by evaluating criteria such as URL paths, request methods, headers, and hostnames. The framework distinguishes itself through its flexible matching capabilities, which include support for regular expressions and dynamic variable extraction from URL segments. It allows developers to organize routes into logical hierarchies, share common path prefixes, and maintain a central registry of named routes to facilitate prog

This project is a VS Code extension that integrates LeetCode problem solving directly into the editor. It functions as a code submission client and competitive programming tool, allowing users to browse, solve, and submit algorithmic challenges without leaving their development environment. The extension distinguishes itself by managing local workspaces and archives, enabling the organization of problem files and descriptions in configurable local folders. It supports multi-account session management and regional endpoint routing to handle different geographic service instances. The tool cov

node-lessons is a comprehensive Node.js programming course and instructional guide. It provides a collection of guided lessons and code examples designed to teach the fundamentals of the Node.js runtime and server-side JavaScript development. The project serves as a practical guide for building web servers and backend applications, specifically covering the implementation of HTTP servers, request routing, and middleware chains. It includes specialized instructional material on managing asynchronous JavaScript workflows through promises and flow control, as well as guides for integrating NoSQL

Sumy is a text summarization library and toolset designed to identify and extract the most important sentences from plain text documents and HTML web pages. It functions as an extractive summarization system, meaning it selects representative original sentences from a source text without generating new words or modifying existing phrasing. The project provides a suite of algorithmic methods for content distillation, including frequency-based sentence scoring and graph-based text ranking. It also includes utilities for HTML-to-text sanitization to isolate primary content and heuristic filterin