How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
bilibili-api is a Bilibili API wrapper and content scraper designed for programmatically accessing video metadata, user profiles, and content data. It functions as an anti-bot crawler framework and a WebSocket live chat client for retrieving platform information and real-time interaction data. The project incorporates tools to bypass anti-crawling measures and rate limits through the use of proxies and TLS fingerprint spoofing. It also includes logic for mapping and converting various video and content identifiers to ensure consistent data retrieval across different endpoints. Its capability
Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
Camoufox is a Firefox-based stealth automation browser designed to evade detection during automated browsing. It combines a fingerprint randomization engine that generates thousands of unique device attributes per session, native-level API interception to spoof WebRTC, WebGL, media, and other fingerprintable properties, and human behavior simulation that moves the cursor along natural, distance-aware trajectories. The browser is compiled from source with build-time stealth patches and runs headlessly via a lightweight virtual display buffer, making it suitable for web scraping, automated testi
curl_cffi is a Python HTTP client built on libcurl that focuses on browser fingerprint impersonation to evade anti-bot detection. By replacing default TLS handshake and HTTP/2 settings with those extracted from real browsers like Chrome and Firefox, it allows HTTP requests that closely mimic actual browser traffic, reducing the likelihood of being blocked by services that fingerprint automated clients.
The main features of lexiforest/curl_cffi are: TLS Fingerprint Impersonators, Anti-Bot Evasion, HTTP Request Clients, Web Scraping Evasion Tools, Request Retries, Connection Pooling, Automated Retry Sessions, Proxy and Fingerprint Rotation.
Open-source alternatives to lexiforest/curl_cffi include: nemo2011/bilibili-api — bilibili-api is a Bilibili API wrapper and content scraper designed for programmatically accessing video metadata,… apify/crawlee — Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction… lwthiker/curl-impersonate. h4ckf0r0day/obscura — Obscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for… daijro/camoufox — Camoufox is a Firefox-based stealth automation browser designed to evade detection during automated browsing. It… nanmicoder/crawlertutorial — CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static…
