Open-source alternatives to Snscrape

Autoscraper is an automatic web scraping library and pattern-based data extractor that learns extraction rules from sample data. It identifies and retrieves text, URLs, and HTML elements from web pages by analyzing sample values to replicate data patterns across different URLs. The system functions as a web scraping model manager, allowing users to save and reload learned rules to maintain consistent data extraction. It supports the export and import of scraping rules to a local file system to avoid repeating the training process for the same website. The library covers automated web data ex

Instaloader is a Python library and command-line utility designed for the automated retrieval, archiving, and analysis of Instagram content. It provides a programmatic interface to fetch media, captions, and metadata from public or private profiles, hashtags, and stories, while maintaining persistent user sessions for authorized access. The tool distinguishes itself through robust archive management and traffic control mechanisms. It supports incremental synchronization, allowing users to resume interrupted downloads and update local collections without redundant requests. To ensure reliable

Twint is an open-source intelligence and data extraction framework designed to gather public social media information. It functions as a command-line utility that retrieves posts, user profiles, and follower lists directly from web interfaces, bypassing the need for official platform developer credentials or authentication keys. The tool distinguishes itself by enabling automated, large-scale data collection through terminal-based orchestration. It supports granular filtering by keywords, geographic locations, time ranges, and account status, allowing researchers to build targeted datasets fo

Social-analyzer is an open-source intelligence framework designed for the automated discovery, correlation, and verification of digital identities across online platforms. It functions as a comprehensive engine for gathering social media intelligence, utilizing distributed browser automation to extract metadata and profile information from hundreds of websites simultaneously. The platform distinguishes itself through its ability to perform cross-platform identity correlation using heuristic-based pattern matching and name permutation generation. It processes these findings through a confidenc

Rengine is an automated reconnaissance framework and vulnerability management platform designed for attack surface monitoring. It functions as a centralized hub for discovering subdomains and open ports, gathering open-source intelligence, and tracking security flaws across target networks. The system integrates large language models to analyze reconnaissance data and generate vulnerability descriptions and insights. It distinguishes itself through a plugin-based tool integration that wraps external security scanning binaries and a target mapping system that tracks changes to assets over time

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It functions as a social media data extractor that simulates browser requests to gather information without the need for official API keys or user account authentication. The tool provides capabilities for gathering public posts, harvesting user profile metadata such as biographies and locations, and retrieving trending topics categorized by geographical region. It can perform targeted content scraping based on specific usernames, hashtags, or search queries. The system manages data

Scraperr is a self-hosted web scraping and crawling platform designed for extracting structured data from websites using XPath selectors. It functions as a containerized system for managing scraping jobs through a queue and analyzing the resulting content using artificial intelligence. The project differentiates itself through its Kubernetes-native architecture, allowing for scalable deployment and management via package managers. It includes a crawling engine capable of domain-level spidering to discover linked pages and a data analyzer that uses artificial intelligence to query extracted we

Twikit is a Python library and API wrapper designed for interacting with X (Twitter). It simulates browser requests and mimics private network traffic to enable programmatic access to the platform without requiring an official API key. The project focuses on social media automation and data extraction, featuring tools for scraping user profiles, trending topics, and chronological tweet histories. It includes a session manager that handles user authentication, two-factor authentication, and cookie persistence to maintain active account access. The library's capabilities cover a broad range of

Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo social network. It functions as a social media data extractor that utilizes a distributed task queue to scale scraping operations across multiple worker nodes. The system includes a graphical administrative interface for configuring crawler settings, target user identifiers, and search keywords. It employs a distributed architecture to increase data throughput and manage large-scale collection of social media content. The tool covers a wide range of data collection capabilities,

WeiboSpider is a social media scraper designed to extract user profiles, posts, and interaction data from the Sina Weibo platform. It functions as a web-based data crawler that retrieves information via external interfaces rather than parsing the visual frontend. The tool includes a content lineage tracer to follow shared posts back to their original sources. It also features a social engagement analyzer to collect view counts and nested comment threads to measure user interaction metrics. The system provides capabilities for keyword-based social monitoring and search result filtering to tra

MechanicalSoup is a Python web automation library and scraping framework designed to simulate browser sessions and navigate websites without requiring JavaScript execution. It functions as an HTML parsing tool and HTTP session manager, allowing for the programmatic retrieval of page content and the automation of web interactions. The library distinguishes itself by combining session persistence with automated form interaction. It maps user data to HTML input fields and selection boxes for programmatic submission and maintains authenticated states by managing cookies and user-agent headers acr

Stealth is a distributed automation engine designed for headless browser orchestration, web scraping, and decentralized network coordination. It functions as a peer-to-peer framework that enables multiple nodes to align automated tasks, share cached web resources, and route traffic through a decentralized proxy architecture. By abstracting the underlying browser implementation, the project provides a unified interface for executing navigation and data extraction workflows across diverse environments. The platform distinguishes itself through its ability to synchronize browser sessions and aut

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

MechanicalSoup is a Python web automation library designed to simulate browser behavior. It functions as a toolkit for web scraping and automation, providing an HTML parsing engine and an HTTP session manager to interact with websites programmatically. The library enables headless web interaction by mimicking a real user session. It manages persistent state through cookie handling and automatic redirect following, allowing for programmatic website navigation and the simulation of complex browser interactions. Its capabilities cover automated form population and submission using CSS selectors

GHunt is a Google account investigator and open-source intelligence framework designed to retrieve publicly available information and metadata associated with Google accounts. It functions as an OSINT data extractor and offensive security framework used to identify user identities and uncover hidden metadata. The tool extracts public profile data from various Google services and exports the findings into structured JSON formats. This allows for the collection and analysis of digital footprints to support security research and reconnaissance.

X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It functions as an HTML data extractor that transforms raw page content into a defined schema using CSS-style selectors. The project implements a headless browser crawler capable of executing JavaScript to render dynamic content. It handles website content discovery through a breadth-first crawling strategy and automatic pagination discovery to traverse multi-page result sets. The framework manages web data pipelines using a concurrency-limited request queue and request rate cont

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data. The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic

This project is a JavaScript client library and SDK for interacting with the Notion REST API. It provides typed methods and utilities to programmatically manage pages, databases, and workspace resources within a Notion environment. The SDK includes tools for implementing OAuth 2.0 authorization flows to grant third-party application access to workspace content. It features mechanisms for handling cursor-based pagination via asynchronous iterators and implements exponential back-off retries to manage rate limits and transient server errors. The library covers a broad range of capabilities, in

ATIO is a AIO Script Developed with Python3. It gathers a total of 23 Discord tools (including a RAT, a Raid Tool, a Token Grabber, a Crash Video Maker, etc). It has a pleasant and intuitive interface to facilitate the use of all with help and explanations for each of them.

LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping

Nitter is a privacy-focused, alternative web interface for viewing public social media content. It functions as a server-side proxy that fetches and renders external posts, allowing users to browse content without requiring a personal account or executing third-party tracking scripts. By stripping away user identifiers and tracking mechanisms, the application provides a lightweight, anonymous viewing experience. The project distinguishes itself through its emphasis on network-level privacy and self-sovereignty. It supports routing traffic through Tor and I2P networks to bypass censorship and

This project is a Model Context Protocol server that acts as an AI workspace connector, bridging large language models to Notion via its REST API. It provides a secure interface for AI assistants to read, write, and manipulate workspace pages, databases, and users. The server utilizes token-efficient markdown serialization to retrieve and update page content, reducing the amount of data processed by language models. It supports multi-user OAuth authentication, allowing the management of multiple Notion accounts within a single deployment by handling authentication tokens on a per-request basi

This project is a comprehensive cybersecurity tool collection designed to support security research, penetration testing, and vulnerability assessment. It functions as a unified penetration testing suite, providing a centralized environment where professionals can access a wide range of offensive security utilities to identify system weaknesses and study attack vectors. The platform distinguishes itself through a modular architecture that aggregates disparate security scripts into a single, hierarchical command-line interface. It simplifies the management of these utilities by integrating ext

This project is a RESTful media extraction service that provides a programmatic interface for downloading video and image content from social media platforms. It functions as a scraper that parses shared URLs and user profile identifiers to isolate direct media streams and associated metadata from platform-specific data structures. The service distinguishes itself through its ability to emulate cryptographic signatures and security tokens required to authenticate requests against protected backend services. By simulating headless browser behavior and managing cookies and headers, the system b

Agent-Reach is an AI agent web gateway and search tool that provides language models with the ability to search and read content from the open web, social media, and community forums without using official APIs. It functions as a routing layer that connects large language models to various internet backends while managing content parsing and connection health. The system enables API-free information retrieval by using open-source backends to extract text and metadata from platforms such as Twitter, Reddit, and YouTube. It converts unstructured website content, RSS feeds, and video transcripts

Th3inspector is a command-line open-source intelligence reconnaissance tool used for gathering public information on websites, phone numbers, and network records. It functions as a central interface for collecting technical metadata and performing various lookups to build profiles of target entities. The project provides specialized verification utilities for validating email addresses, phone numbers, and credit card bank identification numbers. It also includes tools for retrieving domain registration age, ownership records, and identified subdomains from global databases. Additional capabi