Projects sharing features with Curl Cffi

Obscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for AI models to control browser sessions, interact with websites, and extract web data using a WebSocket implementation of the Chrome DevTools Protocol. The project focuses on bot detection evasion by randomizing browser fingerprints, masking native functions, and blocking tracking scripts to mimic human behavior. It further secures identities through a traffic layer that routes network requests via HTTP or SOCKS5 proxies. The system supports large-scale data extraction through

CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage. The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network

cloudscraper is a Python library designed to bypass Cloudflare anti-bot protections by resolving JavaScript challenges and mimicking browser fingerprints. It functions as a specialized tool for accessing websites that employ automated security systems to block scripts and headless browsers. The project differentiates itself through the use of interchangeable JavaScript runtimes, such as Node.js or V8, to execute challenge code and obtain security clearance tokens. It employs a fingerprint rotation engine and HTTP request emulation to rotate browser headers and device identifiers, mimicking hu

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

ai-goofish-monitor is an AI-driven marketplace monitor and containerized web scraper designed to track online listings. It uses multimodal large language models and natural language prompts to analyze product text and images, determining if items meet specific requirements. The system employs an anti-detection workflow that rotates network proxies and authenticated accounts to bypass rate limits. It captures browser cookies and session states to mimic real user behavior during automated requests. The project includes a task scheduler using cron expressions and an embedded SQLite database for

This project is an Amazon web scraper and e-commerce data extractor designed to retrieve product names, prices, and ratings. It functions as a headless browser crawler that converts unstructured web content from product listings into structured JSON and CSV formats. The tool incorporates anti-bot bypass capabilities to circumvent CAPTCHAs and security challenges. It achieves this through the use of residential proxy integration, automatic proxy rotation, and the modification of browser fingerprints to simulate human interaction patterns. The system provides broad web scraping capabilities, i

CyberScraper-2077 is an AI-powered web scraping tool that uses large language models to extract and structure data from websites into organized formats. It functions as an LLM web scraper and AI content parser, transforming unstructured raw web text into specific data schemas. The project distinguishes itself through a suite of anonymity and evasion tools, including proxy rotation, SOCKS-based identity masking, and the ability to route traffic through the Tor network to access hidden onion services. It further includes a bot detection bypass system that employs stealth parameters and custom n

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

This project provides a programmatic client for interacting with AI-powered search services. It is designed to automate research tasks by facilitating batch query processing, real-time response streaming, and document content analysis through both synchronous and asynchronous interfaces. To ensure continuous service access and high throughput, the system includes an automated account provisioning and lifecycle management framework. This allows for the rotation of authenticated sessions and the automatic renewal of credentials, which helps maintain access during high-volume operations. The pr

Rueidis is a high-performance Redis client library for Go that provides a type-safe and asynchronous interface for interacting with Redis servers. It includes a full implementation of the Redis serialization protocol and a dedicated connection manager to handle pooling, multiplexing, and automatic pipelining. The library is distinguished by its support for RDMA connectivity to reduce latency and CPU overhead. It features a distributed lock manager that implements majority-based locking and optimistic concurrency control, as well as client-side caching with invalidation signals to minimize net

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

Apache JMeter is a Java-based performance testing tool and multi-protocol traffic simulator used to analyze the stability and scalability of servers and networks. It functions as a distributed load testing framework that coordinates remote worker nodes from a single controller to generate high volumes of concurrent traffic. The project is distinguished by its ability to simulate traffic across diverse backend systems, including HTTP, JDBC, LDAP, JMS, FTP, and TCP. It provides a headless command-line interface for automated execution and a reporting system that transforms raw sample logs into

This project is a comprehensive Python network request framework designed for both synchronous and asynchronous HTTP communication. It provides a high-performance client capable of executing non-blocking requests within event-driven applications, while also supporting standard blocking calls for simpler scripts. The library is built to operate natively across diverse asynchronous runtimes, automatically detecting and utilizing the underlying event loop for concurrency. What distinguishes this library is its modular architecture, which decouples request construction from network execution thro

This is a Python SDK for interacting with large language models via API. It serves as a client library to generate text, process messages, and manage conversational states, while providing a specialized interface for connecting to models hosted across different cloud infrastructure providers. The SDK includes a tool-calling framework that maps Python functions to JSON schemas, allowing models to execute external tools. It also features a built-in token counting utility to estimate input size before transmission and a server-sent events client for receiving model tokens in real time. The libr

Wretch is a chainable HTTP client library and Fetch API wrapper designed to simplify network request configuration, response parsing, and error handling. It functions as a network request manager and middleware framework for managing the end-to-end lifecycle of HTTP calls. The project is distinguished by a fluent interface for request configuration and a modular extension system. It utilizes a programmable response resolver to automate parsing and a plugin-based architecture to add new capabilities to request and response objects. The library covers a broad range of operational capabilities,

brpc is a high-performance C++ RPC framework and network programming library designed for building distributed systems. It functions as a multi-protocol RPC server capable of hosting and detecting multiple communication protocols, including gRPC, Thrift, HTTP, Redis, and Memcached, on a single TCP port. The project distinguishes itself through high-throughput data transport and memory efficiency, utilizing RDMA-based transport to bypass the kernel TCP stack and zero-copy memory management to eliminate data duplication. It also implements the Raft algorithm for consensus-based state replicatio

Guzzle is a PHP HTTP client used for sending synchronous and asynchronous requests to web services. It serves as a concurrent HTTP request manager, an HTTP stream handler, and a middleware-based HTTP pipeline. The project is a PSR-7 compliant client, utilizing standardized PHP interfaces for requests, responses, and streams. The library differentiates itself through a customizable functional handler stack that allows for the interception and modification of the request and response lifecycle. It features an adapter-based transport system that enables swapping between network implementations,

REALITY is a censorship circumvention tool and network traffic obfuscator designed to bypass internet filters. It functions as a secure tunneling protocol that masks connection patterns and identity by mimicking the TLS handshakes of legitimate target websites. The system removes server-side TLS fingerprints to hide encrypted traffic from network monitoring tools and deep packet inspection. It employs a website mimicking proxy to present valid handshakes and forward traffic to destination servers, masquerading the connection as a legitimate site visit. The project incorporates a quantum-resi

Trojan-go is a network proxy implementation that uses the Trojan protocol to disguise internet traffic as standard TLS to bypass censorship and deep packet inspection. It functions as a tunneling gateway that encapsulates network data within encrypted tunnels to mimic ordinary web browsing. The project distinguishes itself through advanced obfuscation techniques, including TLS fingerprint mimicry to avoid detection by client signature identification and the use of WebSockets to relay encrypted traffic through content delivery networks. It also supports UDP forwarding through the Trojan protoc

dae is a high-performance Linux network tool that functions as an eBPF transparent proxy. It intercepts and redirects packets at the kernel level to route internet traffic based on domains, IP addresses, and process names. The project distinguishes itself by modifying TLS handshakes to simulate browser signatures, which prevents server-side detection of proxy traffic. It also implements a full-cone network address translation gateway to maintain stable bidirectional connections and utilizes a latency-based node selector to automatically route traffic through the fastest available proxy nodes.

nodriver is an asynchronous Chromium browser automation framework that provides headless control and web scraping capabilities. It functions as a Chrome DevTools Protocol client, allowing for granular engine control by attaching directly to the browser's debug port without the need for external driver binaries. The framework is specifically designed as an anti-bot detection bypass tool. It modifies browser fingerprints and protocol headers to evade automated security systems, handle security warnings, and bypass common obstacles like insecure connection alerts. The system covers a broad rang

Undetected-chromedriver is a framework for automated browser navigation designed to bypass anti-bot security measures. It functions by patching browser drivers at the binary level to obscure automation signals, allowing scripts to interact with protected websites without being flagged or blocked by security services. The project distinguishes itself through its ability to maintain stealth during automated sessions, including those executed in headless mode. It achieves this by injecting custom configurations to mimic human user behavior and by hooking into low-level browser debugging protocol

Posting is a terminal-based HTTP request manager designed for developing, testing, and documenting API endpoints. It functions as a keyboard-driven interface that allows users to execute requests, manage collections, and inspect server responses directly within the command line environment. By storing request configurations as plain text files on the local filesystem, the tool ensures that API definitions remain compatible with standard version control systems. The project distinguishes itself by treating API collections as version-controlled assets, enabling users to migrate existing workflo

Karate is a Gherkin API testing framework and JVM test orchestrator designed for API validation, workflow automation, and browser automation. It provides a unified environment for executing scriptable logic on the Java Virtual Machine with thread-safe concurrent processing. The project includes a built-in HTTP mock server to simulate backend API behavior by delivering dynamic responses to intercepted requests. It also features a visual regression testing tool that detects interface changes by comparing screenshots against baseline images using pixel-matching analysis. The system supports API

fake-useragent is a tool for generating realistic browser identification strings and parsing existing agents into structured metadata. It functions as an HTTP user agent generator and a web scraping utility designed to rotate browser identities to mimic different devices during automated data collection. The project provides capabilities for random user-agent generation and filtering based on specific browsers, operating systems, device platforms, or minimum version numbers. It also includes a user agent parser to extract detailed metadata, such as browser versions and device brands, from age