awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
IonicaBizau avatar

IonicaBizau/scrape-it

0
View on GitHub↗
4,074 stars·218 forks·JavaScript·MIT·9 viewsionicabizau.net/blog/30-how-to-write-a-web-scraper-in-node-js↗

Scrape It

scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields.

The tool uses CSS selectors to target specific data points and employs schema-driven data mapping to organize unstructured web text into a consistent format. It supports custom value transformation to convert raw extracted strings into specific data formats.

The system provides capabilities for web data extraction and automated content mapping. It can parse HTML content sourced from URLs, raw HTML strings, or local file system storage, and integrates with headless browsers to process content from dynamic web pages.

Features

  • Web Data Extraction - Programmatically scrapes and processes web content to extract specific information using selectors.
  • Schema-Driven Mappings - Maps CSS selectors to predefined output fields to ensure structured and consistent data extraction.
  • Web Element Mappings - Maps HTML elements to structured data fields using CSS selectors to organize web text.
  • CSS Selector - Uses CSS selectors to target and extract specific data points from the document object model.
  • Web Data Extraction Tools - Provides utilities for scraping, parsing, and structuring information from web pages into usable formats.
  • Data Extractions - Retrieves specific text or attributes from HTML documents using CSS selectors and DOM traversal.
  • HTML Parsers - Converts raw HTML strings or local files into structured trees for programmatic data extraction.
  • Web Scrapers - Functions as a Node.js based tool for extracting structured data from websites and HTML files.
  • Web Scraping and Extraction - Parses HTML and uses selectors to extract structured data from target URLs.
  • HTML Parsing - Extracts structured information from HTML files stored locally or generated by browsers.
  • Local File Operations - Processes raw HTML content directly from local file system storage.
  • Value Transformations - Provides custom functions to transform raw extracted strings into specific data formats.
  • Headless Browsers - Integrates with headless browsers to retrieve content from dynamic or complex web pages.
  • HTML String Parsing - Converts raw HTML strings from files or browser outputs into usable data formats.
  • JavaScript Crawling Frameworks - Human-friendly scraper for Node.js.
  • Web Scraping - Human-friendly web scraping library.

Star history

Star history chart for ionicabizau/scrape-itStar history chart for ionicabizau/scrape-it

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Open-source alternatives to Scrape It

Similar open-source projects, ranked by how many features they share with Scrape It.
  • scrapinghub/portiascrapinghub avatar

    scrapinghub/portia

    9,509View on GitHub↗

    Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

    Python
    View on GitHub↗9,509
  • rchipka/node-osmosisR

    rchipka/node-osmosis

    4,110View on GitHub↗

    This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.

    JavaScript
    View on GitHub↗4,110
  • hickford/mechanicalsouphickford avatar

    hickford/MechanicalSoup

    4,868View on GitHub↗

    MechanicalSoup is a Python web automation library designed to simulate browser behavior. It functions as a toolkit for web scraping and automation, providing an HTML parsing engine and an HTTP session manager to interact with websites programmatically. The library enables headless web interaction by mimicking a real user session. It manages persistent state through cookie handling and automatic redirect following, allowing for programmatic website navigation and the simulation of complex browser interactions. Its capabilities cover automated form population and submission using CSS selectors

    Python
    View on GitHub↗4,868
  • bda-research/node-crawlerbda-research avatar

    bda-research/node-crawler

    6,785View on GitHub↗

    node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

    TypeScriptcheeriocrawlerextract-data
    View on GitHub↗6,785
See all 30 alternatives to Scrape It→

Frequently asked questions

What does ionicabizau/scrape-it do?

scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields.

What are the main features of ionicabizau/scrape-it?

The main features of ionicabizau/scrape-it are: Web Data Extraction, Schema-Driven Mappings, Web Element Mappings, CSS Selector, Web Data Extraction Tools, Data Extractions, HTML Parsers, Web Scrapers.

What are some open-source alternatives to ionicabizau/scrape-it?

Open-source alternatives to ionicabizau/scrape-it include: scrapinghub/portia — Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as… rchipka/node-osmosis — This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow… hickford/mechanicalsoup — MechanicalSoup is a Python web automation library designed to simulate browser behavior. It functions as a toolkit for… bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… nanmicoder/crawlertutorial — CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static… matthewmueller/x-ray — X-ray is a headless browser web scraper and HTML content crawler designed to extract structured data from websites. It…