Awesome GitHub RepositoriesTask Execution Engines

MediaCrawler is an automated web scraping framework designed to extract public posts, comments, and creator metadata from various social media platforms. It functions as a headless browser automator, utilizing real browser instances to render dynamic content and execute the client-side scripts necessary for interacting with modern web interfaces. The system distinguishes itself through a focus on session persistence and network flexibility. It supports remote debugging to reuse active browser sessions and cookies, which helps minimize the risk of triggering platform security challenges. To ma

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

Este proyecto es un framework de rastreo web (web crawler) distribuido y headless Chrome para la extracción de datos. Funciona como un motor de renderizado de JavaScript que utiliza un navegador headless para procesar páginas dinámicas, extrayendo datos estructurados de sitios web que requieren ejecución de JavaScript. El sistema está diseñado para la recolección de datos escalable a través de múltiples nodos, utilizando sincronización de tareas distribuida y cachés compartidas para evitar el trabajo duplicado. Se distingue por la capacidad de emular entornos de cliente específicos configurando user agents y dimensiones de viewport, mientras captura evidencia visual como capturas de pantalla de la página. El framework cubre una gestión integral del rastreo, incluyendo programación de solicitudes con cola de prioridad, recorrido en profundidad y en anchura, y cumplimiento de archivos robots.txt y sitemap.xml. Proporciona herramientas para limitar la concurrencia, monitoreo de eventos y streaming de datos extraídos en formatos CSV o JSON.

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

Go Spider es un framework modular diseñado para construir scrapers web concurrentes y flujos de trabajo de extracción de datos. Proporciona un motor estructurado para orquestar tareas de crawling automatizadas, gestionar la programación de solicitudes y procesar contenido web a través de una tubería unificada. El framework se distingue por una arquitectura altamente configurable que permite a los desarrolladores inyectar lógica personalizada para descargadores, programadores y componentes de almacenamiento a través de contratos basados en interfaces. Gestiona las interacciones de red utilizando limitación de solicitudes basada en middleware y deduplicación de URL, asegurando que las operaciones de crawling permanezcan eficientes y respetuosas con la carga del servidor. El sistema cubre el ciclo de vida completo de la extracción de datos, incluyendo la ejecución concurrente de tareas, el análisis automatizado de varios formatos de contenido y la normalización de codificación de caracteres. También proporciona monitoreo integrado a través de registro de ejecución y rastreo para facilitar la depuración y el análisis de rendimiento. El proyecto se distribuye como una biblioteca para el lenguaje de programación Go.