bjesus/pipet
Pipet
pipet is a command-line tool that turns web scraping into a piped data flow through Unix filters. It provides a set of specialized scrapers — for CSS selector extraction, headless browser JavaScript rendering, JSON API querying, and change monitoring — each outputting structured data that can be transformed by chaining additional commands.
The tool uses declarative selectors (CSS and JSON path expressions) to define what to extract, automatically follows pagination links to collect data across multiple pages, and serializes results into JSON, custom-delimited text, or rendered templates. It can rerun a scraping pipeline on a schedule and trigger a custom command whenever the output changes from the previous run. Headless browser automation allows scraping JavaScript-heavy pages, executing custom scripts, and replicating authenticated sessions by reusing browser request headers.
Additional capabilities include extracting data from HTML pages with nested iterations, querying JSON API endpoints with path syntax, and outputting results in multiple formats. pipet is designed to fit naturally into existing command-line workflows, treating each scraping job as a composable pipe.
Features
- Web Scraping - Extracts structured data from websites using CSS selectors and pipes results to Unix commands.
- Web Page Scraping Extractors - Parses HTML documents using CSS selectors and nested iterations, extracts structured fields, and pipes the output to Unix commands.
- Command Piping - Data moves through a chain of Unix filters where each stage transforms the scraped output.
- Array Iteration Clients - Extracts data from JSON endpoints using path syntax, iterates over arrays, and pipes results to external tools.
- Page Change Monitors - Reruns scraping on a set interval and executes a command when scraped content differs from the previous result.
- Content Diff Monitors - Reruns a scraping pipeline on a schedule and triggers a command when content changes.
- Web Change Monitors - Runs scraping on a schedule and triggers a command when the scraped content differs from the previous result.
- CSS Selectors - Uses CSS selectors to declaratively define which elements to extract from documents.
- Pagination Crawlers - Automatically follows pagination links to scrape data from successive pages.
- Headless Browsers - Controls a headless browser to render JavaScript-heavy pages and execute custom scripts.
- Path Expression Clients - Queries JSON APIs with path expressions and iterates over arrays from the command line.
- Web Scraping Selectors - Scrapes HTML pages using CSS selectors with nested iterations and pagination, outputting structured data to Unix pipes.
- JavaScript Rendering - Navigates pages with a headless browser, executes custom JavaScript, and returns serialized data or triggers UI actions.
- Multi-Format Data Exports - Outputs scraped data in JSON format, plain text with custom separators, or as a rendered template file.
- JSON APIs - Extracts data from JSON APIs using path syntax, iterates over arrays, and pipes results to external tools.
