3 Repos
Awesome GitHub RepositoriesDynamic
Techniques for retrieving data from websites that use JavaScript to render content.
Distinct from Content Extraction: Focuses on extracting data from JS-rendered pages rather than just general raw text extraction.
Explore 3 awesome GitHub repositories matching data & databases · Dynamic. Refine with filters or upvote what's useful.
Awesome Dynamic GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- code4craft/webmagic
code4craft/webmagic
11,680Auf GitHub ansehenWebmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col
Retrieves data from websites using JavaScript to render content, ensuring information is captured from non-static pages.
Javacrawlerframeworkjava - andeya/pholcus
andeya/pholcus
7,578Auf GitHub ansehenPholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The
Uses headless browsers to render JavaScript and extract data from modern, dynamic web pages.
Go - matthewmueller/x-ray
matthewmueller/x-ray
5,904Auf GitHub ansehenX-ray ist ein Headless-Browser-Web-Scraper und HTML-Content-Crawler, der darauf ausgelegt ist, strukturierte Daten von Websites zu extrahieren. Er fungiert als Stream-basierter Data-Scraper und Extraktor, der Selektoren verwendet, um Text und Attribute aus HTML als verschachtelte Objekte oder Arrays abzurufen. Das Projekt enthält einen Request-Rate-Controller zur Steuerung des Netzwerkverkehrs durch Concurrency-Limits, Throttles und Timeouts. Es handhabt das Scrapen dynamischer Websites durch das Rendern von JavaScript via Headless-Browser und führt automatisiertes Website-Crawling mittels Breadth-First-Link-Following und Paginierungs-Management durch. Das System bietet eine Daten-Pipeline, die funktionale Werttransformationen auf Roh-Strings anwendet und Ergebnisse in einen lesbaren Stream schreibt, um Speicherüberläufe bei groß angelegten Web-Scraping-Jobs zu verhindern.
Employs techniques to retrieve data from websites that rely on JavaScript for content rendering.
JavaScript
