bjesus/pipet

Features

• Web Scraping - Extracts structured data from websites using CSS selectors and pipes results to Unix commands.
• Web Page Scraping Extractors - Parses HTML documents using CSS selectors and nested iterations, extracts structured fields, and pipes the output to Unix commands.
• Command Piping - Data moves through a chain of Unix filters where each stage transforms the scraped output.
• Array Iteration Clients - Extracts data from JSON endpoints using path syntax, iterates over arrays, and pipes results to external tools.
• Page Change Monitors - Reruns scraping on a set interval and executes a command when scraped content differs from the previous result.
• Content Diff Monitors - Reruns a scraping pipeline on a schedule and triggers a command when content changes.
• Web Change Monitors - Runs scraping on a schedule and triggers a command when the scraped content differs from the previous result.
• CSS Selectors - Uses CSS selectors to declaratively define which elements to extract from documents.
• Pagination Crawlers - Automatically follows pagination links to scrape data from successive pages.
• Headless Browsers - Controls a headless browser to render JavaScript-heavy pages and execute custom scripts.
• Path Expression Clients - Queries JSON APIs with path expressions and iterates over arrays from the command line.
• Web Scraping Selectors - Scrapes HTML pages using CSS selectors with nested iterations and pagination, outputting structured data to Unix pipes.
• JavaScript Rendering - Navigates pages with a headless browser, executes custom JavaScript, and returns serialized data or triggers UI actions.
• Multi-Format Data Exports - Outputs scraped data in JSON format, plain text with custom separators, or as a rendered template file.
• JSON APIs - Extracts data from JSON APIs using path syntax, iterates over arrays, and pipes results to external tools.

Star history