Projects sharing features with Tiktok Scraper

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

cloudscraper is a Python library designed to bypass Cloudflare anti-bot protections by resolving JavaScript challenges and mimicking browser fingerprints. It functions as a specialized tool for accessing websites that employ automated security systems to block scripts and headless browsers. The project differentiates itself through the use of interchangeable JavaScript runtimes, such as Node.js or V8, to execute challenge code and obtain security clearance tokens. It employs a fingerprint rotation engine and HTTP request emulation to rotate browser headers and device identifiers, mimicking hu

This project is a public proxy aggregator and directory providing curated lists of validated HTTP and SOCKS proxy servers. It features a machine-readable API service and tools designed for anonymous network routing and the automated rotation of outgoing IP addresses. The system distinguishes itself through a proxy rotation tool used to bypass rate limits and prevent detection by automated security systems. It provides a programmatic interface for retrieving and filtering verified proxies by country and protocol, delivering this data in JSON and text formats for integration into custom applica

Annie is a command-line video downloader and web video extraction library written in Go. It functions as a concurrent media downloader designed to fetch video files and playlists from websites via URLs. The tool distinguishes itself through a proxy-aware network layer that supports SOCKS5 and HTTP proxies to bypass regional content restrictions. It also incorporates session cookie integration and referrer spoofing to facilitate the download of authenticated or age-gated content. The project provides capabilities for bulk media acquisition, including batch downloading from text files and extr

AnyCrawl is an AI-powered data extractor, automated web crawler, and headless browser orchestrator. It serves as a web content extraction API and a gateway that connects crawling and scraping tools to language models using a standardized API protocol. The project specializes in converting unstructured website content into structured JSON or markdown optimized for AI assistants. It utilizes language models and JSON schemas to pull specific information into validated formats and provides capabilities for AI page summarization and LLM-optimized content extraction. The system manages comprehensi

Spider_XHS is a data extraction and automation tool built specifically for the Xiaohongshu social platform. It orchestrates multi-step workflows that combine comment tree traversal, cookie-based session reuse, high-resolution media retrieval, keyword search, proxy-backed retries, QR-code login, structured file export, and aggregated user profile collection into a single pipeline. The tool distinguishes itself through its integrated authentication and publishing capabilities, supporting login via QR code scanning or phone verification codes to establish and maintain authenticated sessions. It

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

This project is a LinkedIn data scraper and professional profile extractor designed to collect information from professional networking sites. It functions as a headless browser scraper that extracts professional profiles, company details, and job listings using automated browser sessions. The tool includes a session manager that saves and loads authentication cookies to maintain persistent access to protected profiles. It employs configurable browser settings and user-agent mimicry to simulate human activity and bypass bot detection. Data extraction capabilities cover person profiles, compa

TscanPlus is an external attack surface management tool and security reconnaissance framework designed for discovering network assets, enumerating subdomains, and mapping internet-facing services. It functions as a vulnerability scanning framework and network asset discovery suite to identify security exposure and map active hosts. The platform distinguishes itself by integrating an intelligence layer that uses large language models to analyze raw scan results and identify security weaknesses within JavaScript code. It also includes a dedicated proxy management system that validates and rotat

TikTokDownload is a configurable batch video downloader for TikTok and Douyin that strips watermarks and supports automated downloads from user profiles, likes, and collections. It functions as a social media content archiving tool, enabling users to download videos and audio from these platforms for offline viewing or personal backup. The project distinguishes itself through a modular download pipeline that combines audio extraction, batch scheduling, config-driven workflows, cookie-based authentication, URL parsing, paginated API scraping, and watermark removal. It uses a settings file to c

Lux is a command line video downloader written in Go designed for extracting and saving video and audio from various websites. It functions as a concurrent media downloader that increases transfer speeds by splitting files into fragments and downloading them using multiple threads. The tool serves as a playlist download manager capable of retrieving entire video collections or specific ranges of items. It also operates as a proxy-enabled media client, supporting HTTP and SOCKS5 proxies and session cookies to access region-locked, private, or age-gated content. Additional capabilities include

This project is a Go library and command-line utility designed for the retrieval and local archival of remote video content. It provides a programmatic interface for fetching media streams, allowing users to extract metadata and download video files directly to local storage. The library distinguishes itself through its ability to resolve playback restrictions by performing algorithmic transformations on obfuscated authentication tokens. This signature decryption process enables the tool to bypass standard access limitations, while its interface-driven design allows for the selection of speci

CyberScraper-2077 is an AI-powered web scraping tool that uses large language models to extract and structure data from websites into organized formats. It functions as an LLM web scraper and AI content parser, transforming unstructured raw web text into specific data schemas. The project distinguishes itself through a suite of anonymity and evasion tools, including proxy rotation, SOCKS-based identity masking, and the ability to route traffic through the Tor network to access hidden onion services. It further includes a bot detection bypass system that employs stealth parameters and custom n

Hardseed is a command-line forum media scraper and content archiver. It extracts images and torrent seeds from internet forums in bulk, saving the media and associated metadata to local directories. The tool utilizes a proxy-enabled web scraping engine that rotates network traffic through multiple proxy servers and protocols to bypass rate limits and network restrictions. It includes a keyword-based media filter that matches user-defined strings within entry titles to include or exclude specific topics. The system manages data extraction through batch-processed thread iteration and range-lim

This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It functions as a social media data extractor that simulates browser requests to gather information without the need for official API keys or user account authentication. The tool provides capabilities for gathering public posts, harvesting user profile metadata such as biographies and locations, and retrieving trending topics categorized by geographical region. It can perform targeted content scraping based on specific usernames, hashtags, or search queries. The system manages data

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

Whoogle-search is a self-hosted, containerized metasearch engine designed to provide search results while stripping away advertisements, tracking scripts, and cookies. It functions as a privacy-focused proxy that fetches results from major search providers, ensuring that user activity remains isolated from the original service providers. The platform distinguishes itself through granular traffic management and request-level security. It masks user identity by rotating browser identification strings and routing queries through intermediate proxies. Users can further customize their experience

XDM is a multi-threaded download manager and HTTP download accelerator designed to increase file transfer speeds by splitting files into multiple segments for simultaneous retrieval. It functions as a browser-integrated download client that uses extensions to intercept and capture web download requests. The software includes a streaming video downloader capable of intercepting streaming protocols to save online video content as local files. Additionally, it features a video format converter to change file containers and codecs for compatibility with various devices. The application provides

Livestreamer is a command-line stream extractor and modular streaming client designed to resolve hidden media source URLs from online services. It functions as a media pipeline utility that extracts direct media URLs and binary data for local saving, specialized processing, or routing into external media players. The project uses a plugin-based system with custom heuristic logic to extend service support and override default capabilities. It includes a network proxy router to bypass regional blocks and network restrictions through configurable HTTP and HTTPS proxies, and manages authenticatio

ytdlnis is a mobile application that serves as a graphical client for the yt-dlp engine on Android. It functions as a media downloader and manager, providing a user interface to retrieve video and audio from websites. The project distinguishes itself by integrating directly with the Android system share menu and intents to trigger background downloads from external apps. It includes a dedicated authentication cookie manager to import and sync browser session data, enabling the retrieval of private, age-restricted, or premium content. The application covers broad capability areas including au

This is an HTTP client library used for sending and receiving network requests. It functions as an HTTP traffic replicator, a multipart form uploader, and an OAuth request signer, while also serving as an HTTP client capable of routing traffic through Unix domain sockets for local inter-process communication. The project distinguishes itself with the ability to import and parse HTTP Archive JSON files to reproduce recorded network traffic. It also provides cryptographic OAuth signing to secure API access using hashing algorithms and supports routing requests through Unix domain sockets using

CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage. The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network

Nettacker is an automated penetration testing framework designed to orchestrate reconnaissance, port scanning, and vulnerability detection. It functions as a network reconnaissance tool and vulnerability scanner that identifies open ports, fingerprints services, and checks systems against databases of known security flaws. The framework distinguishes itself by combining a web application crawler for discovering hidden paths via fuzzing with a vulnerability management system that persists scan results in a database to track historical assessments. It also includes specialized capabilities for

This repository contains the comprehensive documentation for a code editor focused on AI-assisted software development and remote development workflows. It covers the implementation of AI agents and language models used for autonomous code generation, large-scale refactoring, and task iteration. The project is distinguished by its deep integration of autonomous AI agents capable of web navigation, application logic validation, and orchestrating multi-step development processes. It provides specialized frameworks for tailoring AI behavior through custom instructions, model context protocols, a

TwitchDownloader is a specialized media retrieval and processing tool designed to archive video-on-demand content and clips from Twitch to local storage. It includes a dedicated chat archiver for fetching and saving chat logs in text or structured data formats. The project features a chat-to-video renderer that transforms raw chat logs into visual overlays and video files, including support for custom emote sets. A media queue manager enables the processing of bulk download lists and the management of multiple concurrent retrieval jobs. The software covers remote content search to find multi

Tartube is a desktop video manager and web video downloader designed for capturing and storing video content from a wide variety of hosting sites for offline viewing. It serves as a graphical interface for the yt-dlp engine to download and organize videos from platforms such as YouTube and Twitch. The application enables web video archiving to prevent data loss from deletions or platform changes. It provides a dedicated interface for categorizing and managing large collections of downloaded web videos within an organized library. The system handles multi-platform video downloading and offlin