bitfield/script
Script
This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It provides a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams.
The library distinguishes itself through a comprehensive toolkit for shell-like operations, including a text processing engine for regular expression filtering and frequency analysis, a filesystem utility toolkit for recursive search and path manipulation, and an integrated HTTP client wrapper for building data pipelines that fetch web content.
The capability surface covers external command execution with template-driven arguments, JSON querying, base64 encoding and cryptographic hashing, and extensive filesystem management including directory enumeration and file existence triggers.
The framework utilizes an interface-based stage composition to facilitate the flow of data between discrete processing steps.
Features
- Concurrent Data Pipelines - Implements a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams.
- Go CLI Frameworks - Provides a comprehensive framework for building structured command-line utilities in Go that process text streams.
- Stream Processing - Implements a stream processing architecture for transforming data through matching, replacement, and extraction.
- Value-Flow Pipelines - Provides a value-flow pipeline architecture where discrete processing stages pass output sequentially as input.
- Text Processing Tools - Includes a text processing engine for regular expression filtering, base64 encoding, and data frequency analysis.
- External Command Execution - Enables running external system programs and capturing their output via single or repeated template-based calls.
- Filesystem Utilities - Provides a comprehensive toolkit for navigating, listing, and manipulating files and directories within Go applications.
- Line-Based Text Filters - Transforms input data line by line using regular expressions and string matching to minimize memory overhead.
- Shell Command Execution - Triggers and controls operating system shell commands while capturing standard output and error streams.
- Shell Scripting Utilities - Enables writing automation scripts in Go using shell-like pipelines and external command execution.
- Go Shell Scripting Frameworks - Provides a framework for writing shell-like scripts in Go with built-in support for piping data and executing external commands.
- Concurrent Task Execution - Implements a concurrent execution model using Go channels to process data streams across multiple pipeline stages.
- Pipeline Stage Interfaces - Uses interface-based composition to allow flexible chaining of filters and sinks within the data pipeline.
- Operational Task Automation - Implements automation for system-level operational tasks, including file searching and executing external binaries.
- Filesystem Object Management - Provides structural filesystem operations including recursive file finding, wildcard listing, and data read/write.
- Data Aggregation Tools - Includes tools for counting input lines and calculating unique entry frequencies to summarize data patterns.
- HTTP Content Retrievers - Provides tools for retrieving web data using GET or POST requests to feed into processing pipelines.
- JSON - Extracts specific information from JSON formatted input using query expressions to isolate required fields.
- Frequency Analyzers - Counts occurrences of unique lines in a stream and sorts them by frequency.
- Pipeline File Inputs - Includes tools for reading file contents, concatenating multiple files, and listing files recursively to feed data into a pipeline.
- Standard Stream Redirection - Captures standard output and error streams from external shell commands to redirect them into internal pipelines.
- Command Piping - Streams content from files or strings into external commands for sequential data processing.
- Command Template Engines - Injects dynamic variables into system command arguments using a templating engine before execution.
- File Path Locators - Includes tools for recursively searching directory structures to locate files matching specific criteria.
- Stream Output Management - Provides mechanisms for writing results to standard output, appending to files, or capturing final output as a string.
- Parametrized Command Execution - Executes specific commands for every line of input by injecting line-based variables into command templates.
- HTTP Data Pipelines - Provides an integrated HTTP client wrapper to fetch web content and pipe responses through filters and transformations.
- HTTP Client Wrappers - Provides an HTTP client wrapper to fetch web content and integrate requests directly into scripting pipelines.
- HTTP Request Clients - Ships a client for executing standard HTTP methods to interact with web resources and pass responses to pipelines.
- Infrastructure Tools - Simplifies shell-like scripting in Go.
- Go Projects - Listed in the “Go Projects” section of the Awesome For Beginners awesome list.
- More to explore - Shell-like scripting library.
