Awesome GitHub RepositoriesData Scraping Tools

This project is a community-maintained, open-source job aggregator that provides a curated database of internship opportunities. It centralizes scattered professional listings into a structured, searchable collection categorized by industry, role, and location to assist students in their career search. The repository distinguishes itself by utilizing a version-controlled data store, where all job listings are maintained as plain text files. This approach enables transparent history tracking and granular change analysis through standard diffing tools. The project relies on an automated data ex

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

This library is a Python-based tool for retrieving historical and real-time financial market data from public sources. It functions as a programmatic interface for downloading stock prices, dividends, financial statements, and corporate calendars, allowing users to perform automated research and analysis on various market assets. The project distinguishes itself by structuring retrieved financial time series directly into tabular data frames, which facilitates mathematical analysis and manipulation of market metrics. It supports efficient data retrieval through multi-threaded batch downloadin

Invidious is a privacy-focused, self-hosted alternative frontend for mainstream video platforms. It operates as a decentralized network of independent instances that provide a lightweight, ad-free interface for consuming media. By acting as a proxy between the user and the content provider, the platform prevents tracking and data collection while maintaining a familiar browsing experience. The project distinguishes itself through its robust suite of network-level traffic management and anonymization tools. It employs techniques such as IP rotation, reverse proxy stream routing, and integratio

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

Hardseed is a command-line forum media scraper and content archiver. It extracts images and torrent seeds from internet forums in bulk, saving the media and associated metadata to local directories. The tool utilizes a proxy-enabled web scraping engine that rotates network traffic through multiple proxy servers and protocols to bypass rate limits and network restrictions. It includes a keyword-based media filter that matches user-defined strings within entry titles to include or exclude specific topics. The system manages data extraction through batch-processed thread iteration and range-lim

WeChatDownload is a content archiving tool for WeChat Official Accounts that enables automated batch downloading of articles, comments, collections, and embedded media assets. It extracts account identifiers and session keys from a single shared article link, then iterates through paginated article lists to retrieve all historical content without requiring separate login credentials. The tool distinguishes itself through its comprehensive capture capabilities, including comment threads, reply chains, and entire article collections alongside the main content. It provides granular control over

InfoSpider is a personal data aggregator and digital footprint analyzer. It extracts user activity and history from social platforms and local browser database files to consolidate information into a unified format. The system functions as a social media archiving tool that converts feed data and albums from external links into downloadable PDF documents for offline preservation. It also serves as a browser history extractor that reads local SQLite database files to retrieve and analyze web navigation history. The project covers capabilities for data aggregation, digital footprint analysis,

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

Horizon ist ein KI-gestütztes Nachrichtenaggregationssystem, das darauf ausgelegt ist, benutzerdefinierte Pipelines zu erstellen, die Informationen aus verschiedenen Webquellen abrufen, filtern und anreichern. Es nutzt Large Language Models, um die Informationsfilterung zu automatisieren und Inhalte zu bewerten, um Rauschen zu entfernen und hochwertige Geschichten hervorzuheben. Das System integriert das Model Context Protocol, um Pipeline-Stufen als Tools für externe KI-Assistenten bereitzustellen. Es verwendet einen einheitlichen Adapter, um verschiedene KI-Modellanbieter für konsistente Inhaltsbewertungs- und Zusammenfassungsaufgaben zu standardisieren. Die Pipeline aggregiert Daten aus RSS-Feeds, sozialen Plattformen, Finanz-Toolkits und Code-Repositories. Sie verwaltet Inhalte durch Deduplizierung, kontingentbasierte Kategoriefilterung und kontextuelle Anreicherung, bevor sie mehrsprachige Briefings per E-Mail, Webhooks oder statischer Website-Bereitstellung liefert. Workflows werden durch wiederkehrende Cloud-Automatisierung orchestriert, um die geplante Sammlung und Bereitstellung verarbeiteter Informationen zu verwalten.

ECommerceCrawlers is an educational collection of Python-based crawler scripts designed to extract data from a variety of public websites, including e-commerce platforms, social media sites, news outlets, and multimedia sources. The project serves as a learning resource for web scraping techniques, offering ready-to-run examples that demonstrate practical data extraction methods. The toolkit covers a broad range of data types, including product listings and prices from online retail platforms, public posts and profiles from social networking sites, articles from news and blogging platforms, p

snscrape ist ein Python-basierter Social-Media-Web-Scraper und Crawler, der darauf ausgelegt ist, öffentliche Posts, Profile und Hashtags aus sozialen Netzwerken ohne die Verwendung offizieller APIs zu extrahieren. Er fungiert als Archivierungstool und Dienstprogramm für die Datensammlung im Bereich Open-Source-Intelligence, was das Sammeln öffentlich verfügbarer Informationen zur Untersuchung von Trends und Personen ermöglicht. Das Tool erleichtert die Extraktion von Social-Media-Daten für Forschungs- und Archivierungszwecke und ermöglicht die Erstellung historischer Aufzeichnungen von Konversationen und Benutzeraktivitäten. Es unterstützt Workflows für akademische soziale Analysen und den Export großer Mengen an Metadaten und Nachrichten in lokale Dateien. Die Funktionen umfassen die Fähigkeit, verschiedene soziale Netzwerke zu scrapen und das Volumen der extrahierten Ergebnisse zu begrenzen. Das System kann entdeckte Elemente als Listen von URLs oder detaillierte Dateien mit Inhalten und Zeitstempeln exportieren.

Dieses Projekt ist ein spezialisierter TikTok-API-Scraper und Datenextraktor. Es fungiert als Proxy-basierter Web-Scraper, der Metadaten von Benutzern, Videobeiträge und Trend-Feeds sammelt und eine Webhook-Datenpipeline bereitstellt, um die gescrapten Informationen via HTTP-Requests an externe URLs weiterzuleiten. Das Tool enthält einen Downloader für Videos ohne Wasserzeichen, der hochauflösende Inhalte im lokalen Speicher ablegt. Es verwendet kryptografische Request-Signierung zur Server-Authentifizierung und nutzt Session-Cookie-Authentifizierung in Kombination mit Proxy-Rotation, um den Netzwerkverkehr zu steuern und Ratenbegrenzungen zu umgehen. Die Funktionen umfassen die Massenextraktion von Inhalten durch Batch-Processing-Pipelines sowie den Abruf von Metadaten für Profile, Hashtags und Musiktitel. Extrahierte Daten werden in strukturierte JSON- oder CSV-Formate exportiert; zudem wird der lokale Status nachverfolgt, um den Download-Verlauf zu protokollieren und doppelte Anfragen zu verhindern.

iScript ist eine Sammlung von Python-Automatisierungsskripten, die für Dateidownloads und Datenextraktion von verschiedenen Webdiensten und Cloud-Plattformen entwickelt wurden. Das Projekt bietet spezialisierte Tools für die Verwaltung von Cloud-Speicher, die Konvertierung von Torrent-Links, das Abrufen von Musik und das Beheben von Archiv-Kodierungsfehlern. Das Toolkit enthält einen Musik-Downloader, der hochwertige Audiotracks abruft und ID3-Metadaten-Tags anwendet, sowie einen Magnet-Link-Konverter, der Torrent-Dateien transformiert und Ergebnisse nach Keywords filtert. Es enthält zudem ein Utility zur Korrektur von Zeichenkodierungsdiskrepanzen in ZIP-Archiven, die unter Windows erstellt wurden, für die Extraktion unter Linux. Zusätzliche Funktionen decken Social-Media-Scraping für Bilder und Videos, Netzwerk-Dateisuche über das eMule-Netzwerk und Cloud-Speicher-Management für rekursive Downloads und Remote-Wiedergabe ab. Zur Verbesserung der Performance verwendet das Projekt Multi-Threaded-Segmented-Downloading und Multi-Process-Content-Fetching.

FxEmbed is a collection of specialized services providing a social media data API, a social media embed gateway, and a URL unshortener and sanitizer. It functions as an edge-deployed content proxy designed to programmatically fetch posts, threads, profiles, and search results from various social platforms. The project transforms social media links into rich media previews and interactive embeds for messaging platforms. It also expands shortened links to their original destinations while removing tracking parameters to improve user privacy and security. The system includes capabilities for so

Spider_XHS is a data extraction and automation tool built specifically for the Xiaohongshu social platform. It orchestrates multi-step workflows that combine comment tree traversal, cookie-based session reuse, high-resolution media retrieval, keyword search, proxy-backed retries, QR-code login, structured file export, and aggregated user profile collection into a single pipeline. The tool distinguishes itself through its integrated authentication and publishing capabilities, supporting login via QR code scanning or phone verification codes to establish and maintain authenticated sessions. It

PRAW is a Python wrapper for the Reddit API, functioning as a REST API client and social media data crawler. It provides a programmatic interface to fetch data, manage account content, and interact with the platform. The library implements a comprehensive OAuth 2.0 client supporting multiple authorization flows, including password, implicit, and refresh tokens, as well as read-only guest access. It distinguishes itself through a rate-limit-aware request scheduler that tracks server-side throttling to prevent API quota exhaustion and employs lazy-loading resource objects to fetch data only whe

This project is an unauthenticated web scraper designed to extract public data from the Twitter frontend API. It functions as a social media data extractor that simulates browser requests to gather information without the need for official API keys or user account authentication. The tool provides capabilities for gathering public posts, harvesting user profile metadata such as biographies and locations, and retrieving trending topics categorized by geographical region. It can perform targeted content scraping based on specific usernames, hashtags, or search queries. The system manages data

Twikit is a Python library and API wrapper designed for interacting with X (Twitter). It simulates browser requests and mimics private network traffic to enable programmatic access to the platform without requiring an official API key. The project focuses on social media automation and data extraction, featuring tools for scraping user profiles, trending topics, and chronological tweet histories. It includes a session manager that handles user authentication, two-factor authentication, and cookie persistence to maintain active account access. The library's capabilities cover a broad range of