Projects sharing features with X Crawl

CyberScraper-2077 is an AI-powered web scraping tool that uses large language models to extract and structure data from websites into organized formats. It functions as an LLM web scraper and AI content parser, transforming unstructured raw web text into specific data schemas. The project distinguishes itself through a suite of anonymity and evasion tools, including proxy rotation, SOCKS-based identity masking, and the ability to route traffic through the Tor network to access hidden onion services. It further includes a bot detection bypass system that employs stealth parameters and custom n

CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage. The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

Scrapegraph-ai is a Python framework that uses large language models to automate the extraction of structured data from websites and documents. It functions as an AI-driven data extraction pipeline that converts unstructured web content into structured formats using natural language processing and graph-based logic. The project utilizes graph-based task orchestration to model scraping workflows as interconnected nodes. It features a pluggable model interface for connecting to cloud or local artificial intelligence providers and can generate executable Python code on the fly to handle site-spe

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data. The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic

Playwright-cli is a command line interface for executing web tasks and automating browser interactions using the Playwright framework. It serves as a browser binary manager for downloading and installing specific browser engines and their required system dependencies, as well as a tool for running automated test suites across multiple engines to verify application behavior. The utility functions as a browser session controller, managing browser profiles and persistent storage states via the command line. It enables the execution of automation suites across different browser engines and config

PhantomJS is a scriptable, headless browser engine based on WebKit that provides a programmatic interface for automating web page interactions. It operates without a graphical user interface, allowing for the execution of JavaScript to navigate pages, manipulate the document object model, and perform functional testing of web applications. The tool distinguishes itself by providing low-level control over the browser rendering lifecycle and network stack. It enables real-time interception and modification of network traffic, alongside the ability to generate visual snapshots and document expor

This is a tool for searching, downloading, and archiving articles and engagement metadata from WeChat official accounts. It functions as a web-based content scraper and data exporter, allowing for the automated retrieval of social media content and the collection of performance metrics. The project distinguishes itself through a system that captures session credentials and authentication cookies from desktop clients via a local proxy to access private engagement data. It utilizes a concurrent proxy-pool fetching mechanism to download large volumes of content while avoiding rate limits, and it

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

PythonSpiderNotes is a comprehensive instructional resource and framework for building web crawlers and extracting data using the Python programming language. It provides a set of methods for parsing unstructured HTML and JSON data into structured formats for persistent storage. The project includes detailed guides and tutorials on browser automation for retrieving dynamic content, as well as a framework for data extraction. It specifically covers anti-bot bypass techniques, such as rotating proxies and spoofing headers, to avoid IP blocks and detection systems. The capability surface extend

Firecrawl is a headless browser automation tool and web crawling engine designed to extract structured data from the web. It functions as an API that transforms raw website content and documents into clean markdown and JSON formats to serve as context for large language models. The project distinguishes itself by using natural language prompts to translate human instructions into targeted data extraction tasks and browser actions. It can execute interactive page navigation, such as clicking and scrolling, and perform automated web research to retrieve structured data without manual interventi

Eko is a framework for designing and deploying agentic workflows, featuring an LLM agent workflow orchestrator and a browser automation engine. It provides a server-side process manager for executing system-level operations and managing local files, alongside a human-in-the-loop agent controller for manual oversight and direction during automated decision processes. The system coordinates multi-agent collaboration through role-based partitioning and workflow orchestration, dividing complex tasks into distinct roles and managing execution handoffs. It integrates the Model Context Protocol to s

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

Magentic-UI is an agentic UI toolkit and framework that enables large language models to interface with real-time browser environments, operating systems, and virtual machines. It provides a sandbox environment where models can execute instructions to manage local files and run shell commands. The project functions as a web interaction orchestrator and browser automation framework, allowing for the execution of end-to-end web workflows and form completions. It coordinates these actions through a system that translates natural language goals into executable sequences. The toolkit covers sever

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

Streamlink is a command line video stream extractor that retrieves direct stream URLs from online services for use in external media players. It functions as a local media stream pipe, redirecting raw video data from web services into local files or players via standard input or HTTP. The project includes a headless browser stream scraper to intercept network requests and extract media data from script-heavy websites, alongside a dedicated processor for HLS and DASH segmented media streams. The tool utilizes a modular video plugin framework, allowing support for new streaming platforms to be

qqbot is a bot framework designed for building automated conversational agents and plugins using the SmartQQ client. It provides a conversational bot SDK for processing messages, managing group memberships, and executing event callbacks. The project is distinguished by its use of a headless browser automator to handle account logins and session management. It also includes an IRC chat bridge that functions as a gateway to forward conversations between a local IRC server and the messaging platform. The framework covers a wide range of capabilities, including dynamic plugin management, time-ba

This project is an automation suite comprising an AI visual asset generator, a browser-based social publisher, an Electron resource extractor, and a Markdown content transformer. It functions as a content automation pipeline that uses large language models to generate text and images for distribution across social media platforms. The system distinguishes itself through specialized visual generation capabilities, producing professional infographics, slide decks, educational comics, and SVG diagrams via structured prompts. It also features a dedicated workflow for extracting resources from Ele

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

This project is an Amazon web scraper and e-commerce data extractor designed to retrieve product names, prices, and ratings. It functions as a headless browser crawler that converts unstructured web content from product listings into structured JSON and CSV formats. The tool incorporates anti-bot bypass capabilities to circumvent CAPTCHAs and security challenges. It achieves this through the use of residential proxy integration, automatic proxy rotation, and the modification of browser fingerprints to simulate human interaction patterns. The system provides broad web scraping capabilities, i

This project provides secure, containerized infrastructure designed for autonomous agents, remote code execution, and cloud development. It functions as a sandboxed environment where AI agents and external processes can execute code, run shell commands, and manage files while remaining isolated from the host system. The system distinguishes itself by implementing the Model Context Protocol, allowing it to act as a standardized tool server that exposes browser and filesystem capabilities to compatible clients. It further integrates headless browser automation, enabling programmatic web navigat

Weibospider is a distributed web crawler designed to extract posts, profiles, and interaction data from the Weibo social network. It functions as a social media data extractor that utilizes a distributed task queue to scale scraping operations across multiple worker nodes. The system includes a graphical administrative interface for configuring crawler settings, target user identifiers, and search keywords. It employs a distributed architecture to increase data throughput and manage large-scale collection of social media content. The tool covers a wide range of data collection capabilities,

This project is a micro-framework designed for building standalone command-line applications and terminal utilities using PHP. It provides a structured environment for developing console tools, incorporating a dependency injection container to manage object lifecycles and a service provider system to handle modular application bootstrapping. The framework distinguishes itself by offering built-in capabilities for compiling applications into self-contained, portable executable binaries. It also includes a centralized task scheduling engine that allows developers to define and execute recurring

Mastra is an orchestration framework designed for building, deploying, and managing autonomous AI agents and multi-agent systems. It provides a comprehensive suite of primitives for creating resilient AI applications, including durable workflow orchestration, event-driven agent loops, and semantic memory management. By integrating these core components, the platform enables developers to build complex, multi-step processes that can reason about goals and execute tasks without manual intervention. The framework distinguishes itself through its focus on observability and secure, isolated execut

Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect, process, and route structured information from websites. It functions as a comprehensive pipeline that manages the entire lifecycle of data gathering, from initial configuration to final storage in external databases or message queues. The platform distinguishes itself through a visual configuration interface that allows users to define extraction rules and manage scraping templates without writing custom code. It supports both static and dynamic content retrieval by integratin