Projects sharing features with InfoSpider

This is a tool for searching, downloading, and archiving articles and engagement metadata from WeChat official accounts. It functions as a web-based content scraper and data exporter, allowing for the automated retrieval of social media content and the collection of performance metrics. The project distinguishes itself through a system that captures session credentials and authentication cookies from desktop clients via a local proxy to access private engagement data. It utilizes a concurrent proxy-pool fetching mechanism to download large volumes of content while avoiding rate limits, and it

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

This platform serves as a centralized dashboard for collecting, visualizing, and analyzing personal activity logs and timestamped life events. It functions as a unified repository that aggregates disparate digital records into a single store, enabling long-term historical tracking and personal data analysis. The system distinguishes itself through a modular report composition engine that groups specific event datasets and visual elements into reusable structures. It incorporates an automated alerting engine that monitors incoming data streams against predefined thresholds, triggering notifica

ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through ad

Maigret is an open-source intelligence framework designed for automated digital footprint discovery and identity investigation. It functions as a search engine that aggregates profile metadata by querying thousands of websites for specific usernames, mapping an individual's online presence across diverse platforms. The tool distinguishes itself through recursive discovery capabilities, which identify links within discovered profiles to expand the scope of an investigation automatically. It supports cross-platform identity correlation by mapping disparate accounts and pseudonymous personas, in

TikTokDownload is a configurable batch video downloader for TikTok and Douyin that strips watermarks and supports automated downloads from user profiles, likes, and collections. It functions as a social media content archiving tool, enabling users to download videos and audio from these platforms for offline viewing or personal backup. The project distinguishes itself through a modular download pipeline that combines audio extraction, batch scheduling, config-driven workflows, cookie-based authentication, URL parsing, paginated API scraping, and watermark removal. It uses a settings file to c

WeChatVideoDownloader is a social media video extractor designed to retrieve and save video content from WeChat Channels for offline local storage. It functions as a content archiver that captures multimedia posts from the WeChat ecosystem to prevent data loss. The tool enables the downloading of video files from social channels, allowing for social media archiving and offline content consumption within different media players.

Delta is a lakehouse table format that brings ACID transactions and data warehouse consistency to large scale data lakes on cloud object storage. It serves as an ACID transaction manager, coordinating atomic commits and serializable isolation for concurrent reads and writes across distributed compute engines. The project provides a multi-engine interoperability layer that uses format translation to allow diverse SQL engines and processing frameworks to read and write the same tables. It functions as a data versioning system, utilizing a transaction log to enable time travel, historical snapsh

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

GhostTrack is an open-source intelligence (OSINT) framework that aggregates geographic, network, and social identity information from public data sources. It functions as a digital footprint analyzer, collecting various pieces of publicly available information to build comprehensive profiles of target individuals. The framework combines multiple investigative capabilities into a single tool, including IP address geolocation, phone number intelligence, and social media username discovery. It distributes queries across external data services to maximize coverage and accuracy, resolving IP addre

OpenAddresses is an open-source geospatial data aggregator and directory that collects public domain and open-license address, parcel, and building datasets from governments and organizations worldwide. It functions as a global index and data warehouse for locating and distributing free geospatial records. The project operates a normalization pipeline that cleans and standardizes diverse source formats into a consistent global coordinate and attribute schema. This process includes a crowdsourced curation pipeline and programmatic quality validation to verify the spatial accuracy and formattin

Snoop is an open-source intelligence framework designed for username scanning, digital footprint analysis, and geospatial intelligence gathering. It functions as a modular scanner that identifies profiles associated with specific usernames and nicknames across thousands of public websites to map a target's online presence. The system features a plugin-based extension architecture that allows for the addition of new platform scrapers and data sources. It provides specialized capabilities for evidence archiving, such as capturing HTML snapshots of discovered profiles and resolving numeric coord

fselect is a command line utility and file metadata query engine that uses SQL-like syntax to filter and find files based on their attributes. It functions as a file system analysis tool capable of selecting files and directories using relational logic, subqueries, and structured data filtering. The tool distinguishes itself by integrating Git tracking status and branch metadata into the search process while respecting project ignore files. It also includes a specialized media metadata search capability that filters audio and image files by technical properties such as resolution, bitrate, an

bqplot is an interactive data visualization library for IPython and Jupyter notebooks that utilizes a grammar of graphics. It functions as a tool for creating 2D charts and maps with real-time updates and bidirectional communication between the kernel and frontend. The library is distinguished by its ability to act as a geographic data visualization tool, rendering choropleth maps and spatial data via GeoJSON and custom projections. It also serves as a financial charting tool for producing OHLC and candle bar charts, and as an interactive dashboard framework for combining plotting widgets wit

This project is a browser rendering service and headless Chrome PDF generator built on Puppeteer. It functions as a backend tool for converting web pages and raw HTML content into PDF documents and screenshots. The service distinguishes itself through browser session control, allowing for the injection of session cookies and the configuration of navigation timeouts to handle authenticated pages. It also includes viewport-based layout scaling to adjust browser dimensions and device scale factors during the rendering process. The broader capability surface covers HTML content export and automa

Flow-dashboard is a centralized personal productivity and knowledge management platform designed to track habits, manage tasks, and aggregate data from disparate sources. It functions as a personal analytics engine that stores records in a relational database, allowing users to visualize behavioral patterns and progress toward long-term goals through detailed charts and reports. The system distinguishes itself through a conversational interface that enables users to interact with their data using natural language commands. By integrating with external services, it synchronizes activity logs,

Airbyte is a data integration platform designed to synchronize information between diverse applications, databases, and data warehouses. It functions as an extract, transform, and load orchestrator that manages automated data movement workflows across cloud, on-premise, and hybrid environments. The platform provides a standardized interface for connectors, enabling the movement of structured and unstructured data while maintaining stateful checkpoints for reliable incremental syncing. The platform distinguishes itself through a containerized architecture that isolates connectors to prevent de

This project is a community-maintained, open-source job aggregator that provides a curated database of internship opportunities. It centralizes scattered professional listings into a structured, searchable collection categorized by industry, role, and location to assist students in their career search. The repository distinguishes itself by utilizing a version-controlled data store, where all job listings are maintained as plain text files. This approach enables transparent history tracking and granular change analysis through standard diffing tools. The project relies on an automated data ex

Social-analyzer is an open-source intelligence framework designed for the automated discovery, correlation, and verification of digital identities across online platforms. It functions as a comprehensive engine for gathering social media intelligence, utilizing distributed browser automation to extract metadata and profile information from hundreds of websites simultaneously. The platform distinguishes itself through its ability to perform cross-platform identity correlation using heuristic-based pattern matching and name permutation generation. It processes these findings through a confidenc

Dompdf is a PHP library that functions as a document rendering engine, transforming HTML and CSS markup into portable document files. It operates by parsing web-based layout attributes and visual properties to generate static documents suitable for reports, invoices, or archival purposes. The library distinguishes itself by integrating a resource-fetching pipeline that retrieves external stylesheets and images to maintain visual fidelity. It also supports the execution of server-side scripts during the document creation process, allowing for the injection of dynamic data and custom logic into

Epoch is a CSS-stylable charting engine and visualization library designed for real-time and statistical data. It functions as a timeseries charting tool that renders historical and live data using a hybrid of SVG and HTML5 Canvas graphics to maintain performance during frequent updates. The library distinguishes itself through a unified CSS querying system that applies styles to both vector and raster plot elements. This allows for visual theme resolution via CSS classes and the ability to customize the appearance of specific data series using stylesheets. The toolset covers a broad range o

html2pdf.js is a client-side JavaScript library used to convert HTML elements and webpages into downloadable PDF documents. It operates entirely within the browser, removing the need for a backend server to generate files. The library employs a conversion pipeline that renders HTML to a canvas before generating the final PDF layout. This process includes the ability to preserve active hyperlinks by overlaying clickable regions on anchor tags and managing page breaks through specific style rules or selectors. Users can configure document appearance through settings for page margins, filenames

Elasticsearch Head is a web-based graphical interface for monitoring and administering Elasticsearch clusters. It serves as a cluster management UI, a topology visualizer for nodes and shards, and a REST API client for sending HTTP requests and analyzing JSON responses. The tool distinguishes itself by providing a visual map of cluster topology to monitor data distribution and health. It includes a local proxy to enable administration of remote clusters that are not directly accessible and supports the injection of basic authentication headers for secure request handling. The platform covers

JimuReport is an open-source reporting and dashboard engine designed to be embedded directly into Spring Boot applications. Its core identity centers on generating data reports and full-screen dashboards from natural language descriptions, eliminating the need for manual design. The platform also provides a conversational query interface that translates plain-language questions into database queries, returning results as tables and charts without requiring SQL knowledge. What distinguishes JimuReport is its integration of AI skills that can be installed with a single command, enabling report

This project is an open-source search data index and a collection of historical search trend data provided as a public trends archive. It serves as an open dataset for analyzing global patterns and events through downloadable files. The repository provides an aggregated index of anonymized and normalized search and media datasets. These resources are designed for academic and professional analysis, allowing for the study of longitudinal trends across different regions and timeframes. The data supports global search trend analysis, market pattern analysis, and public interest research. It ena

This project is a community discussion forum and management suite for developers. It provides a space for technical questioning, resource sharing, and project showcasing, supported by a contributor reward system that uses leaderboards to track and recognize helpful community members. The system includes a migration tool specifically for importing and archiving community conversations from social platforms into GitHub Discussions. It also features a notification bridge that synchronizes forum activity with external chat channels to provide real-time alerts for new posts. The platform supports