Yq | Awesome Repos

This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows.

What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It employs a format-agnostic data model that normalizes diverse inputs, facilitating seamless cross-format conversion and interoperability. The engine supports declarative pipeline execution, allowing users to chain multiple operations through standard input and output streams for automated processing in CI/CD environments.

The tool provides a comprehensive suite of capabilities for data manipulation, including arithmetic and logical evaluations, collection sorting, and temporal arithmetic. It handles advanced tasks such as merging multiple files, splitting documents, and dynamically injecting environment variables or external command output into data fields. Users can also enforce security policies by restricting access to external files or system environment variables during execution.

The software is distributed as a standalone binary, supporting shell completion to assist with command-line productivity.

Features

Command-Line Data Processors - Processes YAML, JSON, XML, and TOML files via a declarative command-line expression syntax.
Data Format Converters - Translates structured data between formats while preserving original comments, styling, and metadata.
Expression-Based Data Querying - Processes structured data by traversing document trees using a domain-specific language to select, filter, and modify nodes.
Data Filtering - Selects elements from collections by evaluating conditions against child nodes to return matching items.
In-Place Data Modifiers - Updates files by modifying the internal node tree while preserving original formatting, comments, and document metadata.
Map Data Structure Manipulation - Manipulates complex data structures through merging, filtering, sorting, and format conversion.
Path-Based Data Access - Updates or removes specific nodes in structured data files using path-based expressions.
File-Based Configuration - Automates the modification of application settings and environment configurations in deployment pipelines.
Data Structure Transformation Techniques - Provides a command-line interface for querying, updating, and converting structured data files.
Data Format Interoperability - Facilitates seamless data translation between YAML, JSON, TOML, and XML formats.
Data Processing Workflows - Automates complex data manipulation and aggregation tasks within shell-based scripting workflows.
Stream Piping Utilities - Facilitates data flow between processes by reading from standard input streams for pipeline integration.
CI/CD Workflows - Executes automated data queries and transformations within CI/CD pipelines.
Infrastructure as Code - Supports infrastructure-as-code maintenance by programmatically updating and validating configuration files.
Abstract Syntax Tree Tools - Parses input files into an intermediate representation to perform structural modifications before serializing back to the target format.
Normalization Engines - Normalizes diverse input formats into a unified internal structure to enable consistent querying and cross-format conversion.
Preservation Utilities - The tool retains original scalar formatting and comments during output processing to maintain the integrity of the source document.
Data Processing - Portable command-line YAML processor.
Data Processing Tools - Query, modify, and merge YAML files with JSON conversion.
TOML and YAML Utilities - Query, modify, and merge YAML files.
Command Line Utilities - Processes YAML and XML files via command line.
Declarative Pipeline Construction - Chains multiple data operations through standard input and output streams to enable complex transformations via shell piping.
Field Exclusion Logic - Removes specific keys or indices from structured data objects while retaining all other elements.
Data Formats - Standardizes file styling and key ordering to ensure consistent data representation across different sources.
Document Processing Tools - Parses and segments multi-document files for structured data extraction.
Structured Data Mergers - Combines the contents of several structured data files into a single output by overlaying or reducing them.
Text String Manipulation Utilities - Manipulates text within structured files using regex, substitution, and case conversion.
Tree Traversal Utilities - Traverses structured data trees to retrieve parent, ancestor, or root nodes relative to a path.
Reference Anchors - The tool assigns anchor names to nodes and creates aliases to existing anchors to maintain references within structured data files.
Data Aggregation Tools - The tool performs mathematical operations, sorting, grouping, and mapping across data collections to produce summarized or restructured output.
Data Formatting Utilities - Applies consistent indentation, colorization, and pretty-printing to improve output readability.
Data Transformation - Transforms rows into columns by rotating data sets for analysis.
Key Existence Verifiers - Checks whether a specific key exists within a map or an index exists within an array.
Structural Data Validators - Verifies that files conform to expected structural requirements by checking tags or types.
Structured Data Extraction - Returns specific ranges of elements from arrays or characters from strings by defining indices.
Structured Comment Management - Adds, updates, and removes line or block comments within structured data files by targeting specific nodes.
Environment Variable Exporters - Converts structured data into shell-compatible key-value assignments for use in external task environments.
External File Importers - Reads content from external files and merges the data directly into the current document structure.
Stream Processing Utilities - Handles large or multi-document files by processing data elements sequentially to minimize memory usage during complex transformations.
External Command Executors - Runs arbitrary shell commands and injects the resulting output into data processing expressions.
Array-to-Map Converters - Transforms arrays into map structures by using indices as keys.
Temporal Arithmetic - The tool adds or subtracts durations from datetime values to calculate future or past timestamps within structured data files.
Environment Variable Injection - Reads system environment variables and inserts them into structured data fields during processing.
Data Operation Pipelines - Sequences multiple data processing expressions so that the output of one operation serves as the input for the next.
File Comparison Tools - Evaluates two files at specific paths and outputs line-by-line differences between matching nodes.
Expression Evaluators - Processes strings as command-line expressions at runtime to construct and execute dynamic logic.
Data Processing Conditionals - The tool combines, negates, or tests sequences of values against logical criteria to determine if data matches specific requirements.
Array Manipulation Utilities - The tool reorders the elements within an array to appear in the opposite sequence from their original position.
Collection Merging - Combines distinct query outputs into a single collection for simultaneous processing.
Data Grouping Utilities - The tool collects multiple expression results into a single array structure to facilitate batch processing or structured output formatting.
Source Tracking - The tool retrieves the filename or numerical index of the document currently being processed to assist in reporting tasks.
XML Parsers - Enables the transformation of structured data into XML with configurable attributes and processing instructions.
Deduplication Utilities - The tool removes redundant elements from a collection while preserving the original sequence of the remaining items.
Data Sorting Engines - The tool reorders arrays or collections based on specific field values or unique identifiers to structure output as required.
Data Type Casting - Forces specific data types for input values to override automatic detection.
Document Splitters - Separates individual items from a collection into distinct documents within a single output stream.
Key Renaming - The tool updates the key of a specific map entry by assigning a new string value to the existing key node.
Path Filters - Retrieves the full traversal path of matching nodes as arrays of keys or indices.
Node Formatting Configurations - Modifies node quoting, indentation, and block styles to ensure output adheres to specific visual standards.
Containerized Execution - Runs data processing tasks within isolated environments by mounting local directories.
Dynamic File Splitters - Writes individual results from a query into separate files by evaluating an expression to generate unique filenames.
Date Formatting Utilities - Parses and reformats date strings between custom patterns, RFC3339, and Unix epoch formats.
Data File Generators - Generates fresh data files from scratch by outputting processed expressions.
Path Variable Resolvers - Uses environment variables to define target paths or keys for data updates and lookups at runtime.
Script Sequences - Executes sequences of update and delete operations defined in structured script files to automate data modifications.
Object Builders - Enables the programmatic construction of new maps and objects from input data.
Array Element Finding - Returns the initial item from an array or the first entry from a map, optionally filtered by expressions.
Arithmetic Operations - The tool calculates the quotient of two numbers or splits a string into multiple segments based on a specified delimiter.
Data Encoding Utilities - Converts text content to and from Base64 format for handling encoded strings.
Fallback Value Logic - The tool returns a specified fallback value when an expression evaluates to null or false to ensure consistent output.
Block Processors - The tool modifies structured data blocks embedded within text files while preserving the surrounding non-structured content.
Data Integrity Verification - Checks files for syntax errors and reports issues to the standard error stream with non-zero exit codes.

antonmedv/fx

20,282View on GitHub

Fx is a command-line processing suite designed for the transformation, conversion, exploration, and visualization of structured data. It functions as a terminal-based utility that handles both automated shell pipelines and interactive navigation of complex, nested data hierarchies. The tool distinguishes itself by integrating a JavaScript-based engine that executes user-provided logic to filter, map, or modify data fields within a sandboxed runtime. It maintains a responsive interface by decoupling data processing from the display loop, allowing users to explore large datasets through an inte

johnkerl/miller

9,911View on GitHub

Miller is a command-line data processor used for filtering, transforming, and aggregating name-indexed tabular data. It functions as a tool for querying and reshaping records across multiple file formats, serving as a converter between CSV, JSON, and YAML. The tool distinguishes itself by using a name-indexed data model, allowing users to manipulate fields by name rather than numeric position. It utilizes single-pass streaming algorithms to compute statistics and summaries on large datasets that exceed available system memory. Its capabilities cover data transformation and analysis, includin

TomWright/dasel

7,976View on GitHub

Dasel is a multi-format data processor and structured data query tool used for querying, editing, and converting data across JSON, YAML, TOML, and XML. It functions as a universal data converter and a cross-format configuration editor that allows users to interact with various serialization formats through a single unified interface. The tool provides a selector-based interface for extracting and searching nested values and modifying keys and values across different file types. It enables the transformation of data between supported structures to change file types while preserving the origina

saulpw/visidata

8,834View on GitHub

VisiData is a terminal-based interactive data analysis tool and browser designed for exploring, filtering, and sorting large tabular datasets. It functions as a structured data inspector that loads and flattens complex formats like JSON, XML, and PCAP into interactive sheets, as well as a terminal file manager for navigating directories and performing staged filesystem operations. The project distinguishes itself by rendering data visualizations, such as scatter plots and histograms, directly in the terminal using Unicode Braille characters. It provides a Python-based data wrangling environme

mikefarahyq

Yq

Features