bisguzar/twitter-scraperArchived
Twitter Scraper
Este proyecto es un scraper web no autenticado diseñado para extraer datos públicos de la API frontend de Twitter. Funciona como un extractor de datos de redes sociales que simula solicitudes de navegador para recopilar información sin necesidad de claves de API oficiales o autenticación de cuenta de usuario.
La herramienta proporciona capacidades para recopilar publicaciones públicas, recolectar metadatos de perfiles de usuario como biografías y ubicaciones, y recuperar temas de tendencia categorizados por región geográfica. Puede realizar scraping de contenido dirigido basado en nombres de usuario, hashtags o consultas de búsqueda específicas.
El sistema gestiona la recolección de datos a través de paginación, emulación de cliente basada en cabeceras para imitar navegadores web y el análisis de respuestas JSON estructuradas desde endpoints públicos.
Features
- Social Media Profile Extractors - Extracts structured public data and profile information from social media platforms through automated scanning.
- Browser Mimicking Requests - Emulates browser headers and tokens to access public API endpoints without authentication.
- Social Media Post Retrievers - Gathers series of public social media posts based on search or user criteria.
- Targeted Content Scraping - Searches for and collects posts based on specific usernames or hashtags for behavioral analysis.
- Social Media Data Scraping - Scrapes public profiles, posts, and trends from social media frontends without requiring OAuth.
- Account Profile Retrievers - Provides utilities for retrieving detailed public user profile metadata, including biographies and locations.
- Public Profile Mapping - Collects biography, location, and birthday data to build detailed public user datasets.
- Cursor-Based Pagination - Uses cursor-based tokens to sequentially retrieve large datasets of posts and profiles.
- Social Trend Analysis - Monitors regional trending topics and hashtags to identify popular public discussions.
- Twitter Data Extraction - Gathers public posts and profile information from Twitter without needing an official API key.
- Twitter Data Extractors - Extracts public posts, user profiles, and trending topics from the Twitter frontend without requiring authentication.
- Social Profile Metadata Extractors - Retrieves public personal metadata such as birthdays and biographies for identity mapping.
- JSON Response Parsers - Provides utilities to convert complex JSON responses from social media APIs into structured data.
- Browser Mimicking Headers - Implements specific HTTP headers to simulate a web browser and evade bot detection.
- User Profile Extractions - Extracts detailed public biographical information, including location and bio, from user profiles.
- User Profile Post Extraction - Retrieves a complete historical list of posts for a specific user profile via paginated scanning.
- Trending Topic Detectors - Detects and retrieves current trending topics for local or regional areas.
- Regional Trend Monitoring - Fetches trending topics for specific geographical areas to monitor local interests.
- Hashtag and Profile Search - Scans multiple pages to collect social media results based on specific hashtags or user profiles.
- Web Scrapers - Fetches and parses live internet content by simulating browser requests to avoid the need for official API keys.
- Dynamic Endpoint Construction - Constructs targeted URL paths based on search terms or user IDs to retrieve specific data categories.
- Query Parameter Configuration - Allows adjustment of API query parameters to filter results by source, location, or user criteria.
