Awesome GitHub RepositoriesBulk Dataset Export

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

FinanceDatabase is a system of data repositories and interfaces providing a corporate fundamental database, a financial market data API, and an SEC filings aggregator. It functions as a financial valuation engine and a macroeconomic indicator feed, offering a programmatic way to access market quotes, corporate fundamentals, and official regulatory disclosures. The project distinguishes itself through an institutional ownership tracker that monitors fund holdings, insider trading activity, and political financial disclosures. It also includes a dedicated tool for extracting and analyzing offic

OSV is a distributed database and aggregator of open-source security advisories that uses a standardized vulnerability schema to track security flaws. It functions as a system for collecting and normalizing security data from diverse ecosystems into a single unified format, providing a web API for querying package vulnerabilities and submitting standardized records. The project distinguishes itself through a security advisory distribution service that supports bulk dataset exports via cloud storage buckets and incremental synchronization of security record updates. It also employs sandbox-bas