NanmiCoder/MediaCrawler
MediaCrawler
MediaCrawler is an automated web scraping framework designed to extract public posts, comments, and creator metadata from various social media platforms. It functions as a headless browser automator, utilizing real browser instances to render dynamic content and execute the client-side scripts necessary for interacting with modern web interfaces.
The system distinguishes itself through a focus on session persistence and network flexibility. It supports remote debugging to reuse active browser sessions and cookies, which helps minimize the risk of triggering platform security challenges. To maintain stable data collection at scale, the tool integrates proxy-based request routing, allowing users to distribute traffic across external IP services to bypass rate limits and geographic restrictions.
The architecture is built for extensibility and modularity, employing a provider pattern that allows developers to integrate new platforms or custom storage backends through standardized interfaces. Users can manage complex scraping workflows via command-line configuration, enabling the definition of specific targets and storage formats—such as JSON, CSV, or various database systems—without modifying the core logic. The project also includes utilities for data visualization, such as generating word clouds from collected comments.
Installation requires setting up the necessary runtime environments, including a JavaScript engine for handling complex client-side rendering and the appropriate browser automation drivers.
Features
- Web Scrapers - Collects posts, comments, and creator details from social platforms using a unified interface.
- Web Scraping Frameworks - Implements automated pipelines for navigating websites and collecting data at scale.
- Browser Automation - Controls real web browsers to render dynamic content and execute client-side scripts.
- Headless Browser Controllers - Manages headless browser instances to navigate dynamic content and bypass security challenges.
- Social Media Scrapers - Automates browser interaction to extract posts, comments, and creator metadata from social platforms.
- Browser Session Persistence - Maintains persistent login states to minimize detection and avoid repetitive security challenges.
- Media Crawlers - Automates media retrieval across various social platforms using command-line configuration.
- Task Execution Engines - Retrieves specific content or user contributions by running crawling tasks across supported platforms.
- Automation Scripts - Automates scraping tasks via command-line arguments and configuration files.
- Proxy Management - Distributes network traffic across external IP services to circumvent rate limits and access restricted content.
- Proxy Management Services - Routes network requests through external proxies to bypass rate limits and geo-blocking.
- Data Exporters - Saves collected information into formats like CSV, JSON, SQLite, MySQL, or MongoDB.
- Data Storage Adapters - Persists data through interchangeable drivers that abstract the underlying database implementation.
- Proxy-Aware Clients - Routes traffic through external services to manage rate limits and access geo-restricted content.
- Social Media Extraction Tools - Extracts public posts, comments, and creator metadata from various social platforms.
- Data Aggregation Pipelines - Standardizes data retrieval from multiple services into a unified format for consistent processing.
- Configuration Management - Decouples task logic from the runtime environment using external configuration files and command-line arguments.
- Proxy Configuration Tools - Configures network requests through external proxy services to bypass rate limits.
- Remote Debugging Tools - Connects to existing browser instances to reuse cookies and login sessions.
- Session Management - Attaches to existing browser instances to reuse active cookies and login sessions.
