awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
ruipgil avatar

ruipgil/scraperjs

0
View on GitHub↗

Scraperjs

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data.

The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic before data extraction.

The tool covers a broad range of capabilities, including dynamic and static content extraction, multi-site scraping workflows via URL pattern mapping, and the coordination of HTTP requests through promise-based lifecycle chains.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Automated Web Data Collection - Converts unstructured web content into structured data through programmable sequences of requests and parsing.
  • Headless Browser Automation - Provides a framework for programmatically controlling headless browsers to render dynamic content and extract data.
  • Automated Web Scraping - Automates the collection of structured information from websites using an asynchronous workflow.
  • Content Extraction - Parses static HTML using selectors to retrieve specific elements and attributes.
  • DOM-Based Extractions - Transforms unstructured web content into usable data by reading DOM elements via selector-based extraction.
  • Structured Web Data Parsing - Parses content from both static and dynamic web pages to collect structured information.
  • Web Data Scraping - Extracts data from JavaScript-heavy websites by rendering pages in a headless browser.
  • Live Page Content Extraction - Retrieves elements and attributes from live or static web pages using a selector-based interface.
  • Browser Automation - Provides programmatic control of browsers to manipulate the DOM and extract information.
  • Scraping Workflow Orchestrators - Acts as a promise-based manager for sequencing HTTP requests, timeouts, and URL routing during data collection.
  • DOM Selectors - Uses CSS and XPath selectors to extract structured data and attributes from HTML elements.
  • Dynamic Content Extraction - Renders JavaScript-heavy websites in a headless browser to retrieve content that requires client-side execution.
  • Headless Browser Orchestration - Provides a system for managing headless browser instances to render dynamic content and execute JavaScript across multiple tasks.
  • Web Scraping and Extraction - Implements a programmable interface for parsing HTML and extracting structured data from both static and dynamic websites.
  • Multi-Site Scraping Workflows - Maps different URL patterns to unique extraction logic to traverse and scrape multiple website structures simultaneously.
  • Regex-Based Route Matching - Uses regular expressions to match URL patterns and apply specific scraping logic to different website structures.
  • URL Pattern Matching - Maps specific URL patterns to unique scraping logic to traverse complex website structures.
  • Promise-Based Chains - Implements asynchronous promise chains to sequence HTTP requests and data extraction tasks.
  • Browser Instance Pooling - Optimizes resource usage by sharing a single headless browser instance across multiple scraping sessions.
  • Scraper Lifecycles - Orchestrates sequences of requests and extraction tasks using a promise chain with integrated timeout and status triggers.
  • Instance Pooling - Optimizes resource usage by sharing a single headless browser instance across multiple scraping tasks.
  • Script Injections - Allows the injection of custom JavaScript into the browser environment to manipulate the DOM before extraction.
  • JavaScript Crawling Frameworks - Versatile web scraper for Node.js.
  • Web Scraping - Versatile web scraping utility.
3,718 stars·183 forks·JavaScript·MIT·26 views

Star history

Star history chart for ruipgil/scraperjsStar history chart for ruipgil/scraperjs

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does ruipgil/scraperjs do?

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data.

What are the main features of ruipgil/scraperjs?

The main features of ruipgil/scraperjs are: Automated Web Data Collection, Headless Browser Automation, Automated Web Scraping, Content Extraction, DOM-Based Extractions, Structured Web Data Parsing, Web Data Scraping, Live Page Content Extraction.

Which projects share features with ruipgil/scraperjs?

Projects with overlapping indexed features include: huaying/instagram-crawler — This project is a web scraping and automation tool designed to collect public data from Instagram and perform… rchipka/node-osmosis — This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow… pyppeteer/pyppeteer — Pyppeteer is a Python library for controlling Chromium-based browsers using the Chrome DevTools Protocol. It functions… nanmicoder/crawlertutorial — CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static… browser-act/skills — This project provides an agentic web interaction engine designed to facilitate autonomous browser automation and… apify/crawlee — Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction…

Projects sharing features with Scraperjs

These projects share indexed features with Scraperjs. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • huaying/instagram-crawlerhuaying avatar

    huaying/instagram-crawler

    1,335View on GitHub↗

    This project is a web scraping and automation tool designed to collect public data from Instagram and perform automated social media interactions. It functions by gathering profile details, captions, media files, and engagement metrics directly from web pages, bypassing the need for official developer interfaces or platform-specific credentials. The tool distinguishes itself by combining data extraction with automated engagement capabilities. It allows users to programmatically interact with content by liking posts that match specific search criteria or hashtags, aiming to increase account vi

    Pythonautoautolikerinstagram
    View on GitHub↗1,335
  • rchipka/node-osmosisR

    rchipka/node-osmosis

    4,110View on GitHub↗

    This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.

    JavaScript
    View on GitHub↗4,110
  • nanmicoder/crawlertutorialNanmiCoder avatar

    NanmiCoder/CrawlerTutorial

    4,262View on GitHub↗

    CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage. The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network

    Python
    View on GitHub↗4,262
  • pyppeteer/pyppeteerpyppeteer avatar

    pyppeteer/pyppeteer

    3,948View on GitHub↗

    Pyppeteer is a Python library for controlling Chromium-based browsers using the Chrome DevTools Protocol. It functions as a headless browser automation tool, allowing for the programmatic navigation of web pages and the extraction of data from dynamic websites. The project provides low-level browser control through direct communication with the Chrome DevTools Protocol, enabling the interception and modification of network traffic. It differentiates itself by offering specialized performance profiling capabilities, including the measurement of JavaScript and CSS code coverage and the capture

    Python
    View on GitHub↗3,948
Compare all 30 related projects→