Awesome GitHub RepositoriesTask Execution Engines

MediaCrawler is an automated web scraping framework designed to extract public posts, comments, and creator metadata from various social media platforms. It functions as a headless browser automator, utilizing real browser instances to render dynamic content and execute the client-side scripts necessary for interacting with modern web interfaces. The system distinguishes itself through a focus on session persistence and network flexibility. It supports remote debugging to reuse active browser sessions and cookies, which helps minimize the risk of triggering platform security challenges. To ma

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

This project is a distributed headless Chrome web crawler and data extraction framework. It functions as a JavaScript rendering engine that uses a headless browser to process dynamic pages, extracting structured data from websites that require JavaScript execution. The system is designed for scalable data collection across multiple nodes, using distributed task synchronization and shared caches to prevent duplicate work. It distinguishes itself through the ability to emulate specific client environments by configuring user agents and viewport dimensions, while capturing visual evidence such a

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

Go Spider ist ein modulares Framework für den Aufbau konkurrierender Web-Scraper und Datenextraktions-Workflows. Es bietet eine strukturierte Engine zur Orchestrierung automatisierter Crawling-Aufgaben, zur Verwaltung der Request-Planung und zur Verarbeitung von Webinhalten durch eine einheitliche Pipeline. Das Framework zeichnet sich durch eine hochgradig konfigurierbare Architektur aus, die es Entwicklern ermöglicht, benutzerdefinierte Logik für Downloader, Scheduler und Speicherkomponenten über Interface-basierte Verträge einzubinden. Es verwaltet Netzwerkinteraktionen mittels Middleware-basiertem Request-Throttling und URL-Deduplizierung, um sicherzustellen, dass Crawling-Operationen effizient bleiben und die Serverlast respektieren. Das System deckt den gesamten Lebenszyklus der Datenextraktion ab, einschließlich konkurrierender Aufgabenausführung, automatisiertem Parsing verschiedener Inhaltsformate und Normalisierung der Zeichenkodierung. Zudem bietet es integriertes Monitoring durch Ausführungs-Logging und Tracing, um Debugging und Performance-Analyse zu erleichtern. Das Projekt wird als Bibliothek für die Programmiersprache Go verteilt.