# mikefarah/yq

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/mikefarah-yq).**

14,913 stars · 744 forks · Go · mit

## Links

- GitHub: https://github.com/mikefarah/yq
- Homepage: https://mikefarah.gitbook.io/yq/
- awesome-repositories: https://awesome-repositories.com/repository/mikefarah-yq.md

## Topics

`bash` `cli` `csv` `devops-tools` `golang` `hcl` `json` `portable` `properties` `splat` `terraform` `toml` `xml` `yaml` `yaml-processor`

## Description

This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows.

What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It employs a format-agnostic data model that normalizes diverse inputs, facilitating seamless cross-format conversion and interoperability. The engine supports declarative pipeline execution, allowing users to chain multiple operations through standard input and output streams for automated processing in CI/CD environments.

The tool provides a comprehensive suite of capabilities for data manipulation, including arithmetic and logical evaluations, collection sorting, and temporal arithmetic. It handles advanced tasks such as merging multiple files, splitting documents, and dynamically injecting environment variables or external command output into data fields. Users can also enforce security policies by restricting access to external files or system environment variables during execution.

The software is distributed as a standalone binary, supporting shell completion to assist with command-line productivity.

## Tags

### Development Tools & Productivity

- [Command-Line Data Processors](https://awesome-repositories.com/f/development-tools-productivity/command-line-data-processors.md) — Processes YAML, JSON, XML, and TOML files via a declarative command-line expression syntax.
- [Stream Piping Utilities](https://awesome-repositories.com/f/development-tools-productivity/stream-piping-utilities.md) — Facilitates data flow between processes by reading from standard input streams for pipeline integration. ([source](https://mikefarah.gitbook.io/yq/commands/evaluate.md))
- [Text String Manipulation Utilities](https://awesome-repositories.com/f/development-tools-productivity/developer-utilities-libraries/programming-logic-libraries/text-string-manipulation-utilities.md) — Manipulates text within structured files using regex, substitution, and case conversion. ([source](https://mikefarah.gitbook.io/yq/operators/string-operators.md))
- [Structured Comment Management](https://awesome-repositories.com/f/development-tools-productivity/comment-formatting-utilities/structured-comment-management.md) — Adds, updates, and removes line or block comments within structured data files by targeting specific nodes. ([source](https://mikefarah.gitbook.io/yq/operators/comment-operators.md))
- [Environment Variable Exporters](https://awesome-repositories.com/f/development-tools-productivity/environment-variable-exporters.md) — Converts structured data into shell-compatible key-value assignments for use in external task environments. ([source](https://mikefarah.gitbook.io/yq/usage/shellvariables.md))
- [External File Importers](https://awesome-repositories.com/f/development-tools-productivity/project-imports/external-file-importers.md) — Reads content from external files and merges the data directly into the current document structure. ([source](https://mikefarah.gitbook.io/yq/operators/load.md))
- [Stream Processing Utilities](https://awesome-repositories.com/f/development-tools-productivity/stream-processing-utilities.md) — Handles large or multi-document files by processing data elements sequentially to minimize memory usage during complex transformations.
- [Node Formatting Configurations](https://awesome-repositories.com/f/development-tools-productivity/code-quality-analysis/static-analysis-engines/static-analysis-tools/code-formatting-tools/quote-style-configurations/node-formatting-configurations.md) — Modifies node quoting, indentation, and block styles to ensure output adheres to specific visual standards. ([source](https://mikefarah.gitbook.io/yq/v3.x/commands/write-update.md))
- [Containerized Execution](https://awesome-repositories.com/f/development-tools-productivity/command-execution/containerized-execution.md) — Runs data processing tasks within isolated environments by mounting local directories. ([source](https://mikefarah.gitbook.io/yq/v3.x/master.md))
- [Dynamic File Splitters](https://awesome-repositories.com/f/development-tools-productivity/configuration-generators/configuration-file-generators/dynamic-file-splitters.md) — Writes individual results from a query into separate files by evaluating an expression to generate unique filenames. ([source](https://mikefarah.gitbook.io/yq/usage/split-into-multiple-files.md))
- [Date Formatting Utilities](https://awesome-repositories.com/f/development-tools-productivity/date-formatting-utilities.md) — Parses and reformats date strings between custom patterns, RFC3339, and Unix epoch formats. ([source](https://mikefarah.gitbook.io/yq/operators/datetime.md))
- [Data File Generators](https://awesome-repositories.com/f/development-tools-productivity/documentation-generators/data-file-generators.md) — Generates fresh data files from scratch by outputting processed expressions. ([source](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks.md))
- [Path Variable Resolvers](https://awesome-repositories.com/f/development-tools-productivity/dynamic-variable-evaluators/path-variable-resolvers.md) — Uses environment variables to define target paths or keys for data updates and lookups at runtime. ([source](https://mikefarah.gitbook.io/yq/operators/env-variable-operators.md))

### Data & Databases

- [Data Format Converters](https://awesome-repositories.com/f/data-databases/data-format-converters.md) — Translates structured data between formats while preserving original comments, styling, and metadata.
- [Expression-Based Data Querying](https://awesome-repositories.com/f/data-databases/expression-based-data-querying.md) — Processes structured data by traversing document trees using a domain-specific language to select, filter, and modify nodes.
- [Data Filtering](https://awesome-repositories.com/f/data-databases/data-filtering.md) — Selects elements from collections by evaluating conditions against child nodes to return matching items. ([source](https://mikefarah.gitbook.io/yq/v3.x/usage/path-expressions.md))
- [In-Place Data Modifiers](https://awesome-repositories.com/f/data-databases/in-place-data-modifiers.md) — Updates files by modifying the internal node tree while preserving original formatting, comments, and document metadata.
- [Map Data Structure Manipulation](https://awesome-repositories.com/f/data-databases/map-data-structure-manipulation.md) — Manipulates complex data structures through merging, filtering, sorting, and format conversion.
- [Path-Based Data Access](https://awesome-repositories.com/f/data-databases/path-based-data-access.md) — Updates or removes specific nodes in structured data files using path-based expressions. ([source](https://mikefarah.gitbook.io/yq/operators/path.md))
- [Data Format Interoperability](https://awesome-repositories.com/f/data-databases/data-format-interoperability.md) — Facilitates seamless data translation between YAML, JSON, TOML, and XML formats.
- [Data Processing Workflows](https://awesome-repositories.com/f/data-databases/data-processing-workflows.md) — Automates complex data manipulation and aggregation tasks within shell-based scripting workflows.
- [Declarative Pipeline Construction](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing-frameworks/declarative-pipeline-construction.md) — Chains multiple data operations through standard input and output streams to enable complex transformations via shell piping.
- [Field Exclusion Logic](https://awesome-repositories.com/f/data-databases/data-query-filters/field-exclusion-logic.md) — Removes specific keys or indices from structured data objects while retaining all other elements. ([source](https://mikefarah.gitbook.io/yq/operators/omit.md))
- [Data Formats](https://awesome-repositories.com/f/data-databases/data-serialization-formats/data-formats.md) — Standardizes file styling and key ordering to ensure consistent data representation across different sources. ([source](https://mikefarah.gitbook.io/yq/usage/tips-and-tricks.md))
- [Document Processing Tools](https://awesome-repositories.com/f/data-databases/document-processing-tools.md) — Parses and segments multi-document files for structured data extraction. ([source](https://mikefarah.gitbook.io/yq/v3.x/commands/read.md))
- [Structured Data Mergers](https://awesome-repositories.com/f/data-databases/structured-data-mergers.md) — Combines the contents of several structured data files into a single output by overlaying or reducing them. ([source](https://mikefarah.gitbook.io/yq/readme.md))
- [Data Aggregation Tools](https://awesome-repositories.com/f/data-databases/data-aggregation-tools.md) — The tool performs mathematical operations, sorting, grouping, and mapping across data collections to produce summarized or restructured output. ([source](https://mikefarah.gitbook.io/yq/operators.md))
- [Data Formatting Utilities](https://awesome-repositories.com/f/data-databases/data-formatting-utilities.md) — Applies consistent indentation, colorization, and pretty-printing to improve output readability. ([source](https://mikefarah.gitbook.io/yq/v3.x/usage/output-format.md))
- [Data Transformation](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation.md) — Transforms rows into columns by rotating data sets for analysis. ([source](https://mikefarah.gitbook.io/yq/operators/pivot.md))
- [Key Existence Verifiers](https://awesome-repositories.com/f/data-databases/key-value-stores/key-listing/key-existence-verifiers.md) — Checks whether a specific key exists within a map or an index exists within an array. ([source](https://mikefarah.gitbook.io/yq/operators/has.md))
- [Structural Data Validators](https://awesome-repositories.com/f/data-databases/structural-data-validators.md) — Verifies that files conform to expected structural requirements by checking tags or types. ([source](https://mikefarah.gitbook.io/yq/upgrading-from-v3.md))
- [Structured Data Extraction](https://awesome-repositories.com/f/data-databases/structured-data-extraction.md) — Returns specific ranges of elements from arrays or characters from strings by defining indices. ([source](https://mikefarah.gitbook.io/yq/operators/slice-array.md))
- [Array Manipulation Utilities](https://awesome-repositories.com/f/data-databases/array-manipulation-utilities.md) — The tool reorders the elements within an array to appear in the opposite sequence from their original position. ([source](https://mikefarah.gitbook.io/yq/operators/reverse.md))
- [Collection Merging](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-extraction-ingestion/data-import-and-export/csv-import-managers/collection-merging.md) — Combines distinct query outputs into a single collection for simultaneous processing. ([source](https://mikefarah.gitbook.io/yq/operators/union.md))
- [Data Grouping Utilities](https://awesome-repositories.com/f/data-databases/data-grouping-utilities.md) — The tool collects multiple expression results into a single array structure to facilitate batch processing or structured output formatting. ([source](https://mikefarah.gitbook.io/yq/operators/collect-into-array.md))
- [Source Tracking](https://awesome-repositories.com/f/data-databases/data-ingestion-sources/source-metadata-capture/source-tracking.md) — The tool retrieves the filename or numerical index of the document currently being processed to assist in reporting tasks. ([source](https://mikefarah.gitbook.io/yq/operators/file-operators.md))
- [XML Parsers](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-serialization/xml-parsers.md) — Enables the transformation of structured data into XML with configurable attributes and processing instructions. ([source](https://mikefarah.gitbook.io/yq/usage/xml.md))
- [Deduplication Utilities](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/processing-pipelines/item-pipelines/data-item-filters/deduplication-utilities.md) — The tool removes redundant elements from a collection while preserving the original sequence of the remaining items. ([source](https://mikefarah.gitbook.io/yq/operators/unique.md))
- [Data Sorting Engines](https://awesome-repositories.com/f/data-databases/data-sorting-engines.md) — The tool reorders arrays or collections based on specific field values or unique identifiers to structure output as required. ([source](https://mikefarah.gitbook.io/yq/recipes.md))
- [Data Type Casting](https://awesome-repositories.com/f/data-databases/data-type-casting.md) — Forces specific data types for input values to override automatic detection. ([source](https://mikefarah.gitbook.io/yq/v3.x/usage/value-parsing.md))
- [Document Splitters](https://awesome-repositories.com/f/data-databases/document-splitters.md) — Separates individual items from a collection into distinct documents within a single output stream. ([source](https://mikefarah.gitbook.io/yq/operators/split-into-documents.md))
- [Key Renaming](https://awesome-repositories.com/f/data-databases/key-value-stores/atomic-key-updaters/key-renaming.md) — The tool updates the key of a specific map entry by assigning a new string value to the existing key node. ([source](https://mikefarah.gitbook.io/yq/operators/keys.md))
- [Path Filters](https://awesome-repositories.com/f/data-databases/path-filters.md) — Retrieves the full traversal path of matching nodes as arrays of keys or indices. ([source](https://mikefarah.gitbook.io/yq/operators/path.md))

### DevOps & Infrastructure

- [File-Based Configuration](https://awesome-repositories.com/f/devops-infrastructure/configuration-management/file-based-configuration.md) — Automates the modification of application settings and environment configurations in deployment pipelines.
- [CI/CD Workflows](https://awesome-repositories.com/f/devops-infrastructure/ci-cd-workflows.md) — Executes automated data queries and transformations within CI/CD pipelines. ([source](https://mikefarah.gitbook.io/yq/usage/github-action.md))
- [Infrastructure as Code](https://awesome-repositories.com/f/devops-infrastructure/infrastructure/infrastructure-as-code.md) — Supports infrastructure-as-code maintenance by programmatically updating and validating configuration files.
- [External Command Executors](https://awesome-repositories.com/f/devops-infrastructure/automation-orchestration/task-execution-frameworks/external-command-executors.md) — Runs arbitrary shell commands and injects the resulting output into data processing expressions. ([source](https://mikefarah.gitbook.io/yq/operators/system-operators.md))
- [Script Sequences](https://awesome-repositories.com/f/devops-infrastructure/automation-orchestration/task-execution-frameworks/automation-frameworks/action-execution/script-sequences.md) — Executes sequences of update and delete operations defined in structured script files to automate data modifications. ([source](https://mikefarah.gitbook.io/yq/v3.x/upgrading-from-v2.md))

### Education & Learning Resources

- [Data Structure Transformation Techniques](https://awesome-repositories.com/f/education-learning-resources/algorithm-and-data-structure-guides/data-structure-transformation-techniques.md) — Provides a command-line interface for querying, updating, and converting structured data files.
- [Tree Traversal Utilities](https://awesome-repositories.com/f/education-learning-resources/technical-domain-education/technical-academic-domains/algorithmic-design-analysis/tree-data-structures/tree-traversal-utilities.md) — Traverses structured data trees to retrieve parent, ancestor, or root nodes relative to a path. ([source](https://mikefarah.gitbook.io/yq/operators/parent.md))
- [Object Builders](https://awesome-repositories.com/f/education-learning-resources/curricula-instructional-design/curricula-roadmaps/foundations-study-skills/foundational-computer-science-modules/data-structures-and-algorithms/data-structure-transformations/object-builders.md) — Enables the programmatic construction of new maps and objects from input data. ([source](https://mikefarah.gitbook.io/yq/operators/create-collect-into-object.md))

### Software Engineering & Architecture

- [Abstract Syntax Tree Tools](https://awesome-repositories.com/f/software-engineering-architecture/abstract-syntax-tree-tools.md) — Parses input files into an intermediate representation to perform structural modifications before serializing back to the target format.
- [Normalization Engines](https://awesome-repositories.com/f/software-engineering-architecture/data-formats/normalization-engines.md) — Normalizes diverse input formats into a unified internal structure to enable consistent querying and cross-format conversion.
- [Preservation Utilities](https://awesome-repositories.com/f/software-engineering-architecture/data-formats/preservation-utilities.md) — The tool retains original scalar formatting and comments during output processing to maintain the integrity of the source document. ([source](https://mikefarah.gitbook.io/yq/v3.x/usage/output-format.md))
- [Environment Variable Injection](https://awesome-repositories.com/f/software-engineering-architecture/application-lifecycle-management/configuration-management/environment-variable-management/environment-variable-injection.md) — Reads system environment variables and inserts them into structured data fields during processing. ([source](https://mikefarah.gitbook.io/yq/operators/env-variable-operators.md))
- [Data Operation Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/data-operation-pipelines.md) — Sequences multiple data processing expressions so that the output of one operation serves as the input for the next. ([source](https://mikefarah.gitbook.io/yq/operators/pipe.md))
- [File Comparison Tools](https://awesome-repositories.com/f/software-engineering-architecture/file-comparison-tools.md) — Evaluates two files at specific paths and outputs line-by-line differences between matching nodes. ([source](https://mikefarah.gitbook.io/yq/v3.x/commands/compare.md))
- [Fallback Value Logic](https://awesome-repositories.com/f/software-engineering-architecture/default-configuration-values/fallback-value-logic.md) — The tool returns a specified fallback value when an expression evaluates to null or false to ensure consistent output. ([source](https://mikefarah.gitbook.io/yq/operators/alternative-default-value.md))
- [Block Processors](https://awesome-repositories.com/f/software-engineering-architecture/inline-data-embedding/block-processors.md) — The tool modifies structured data blocks embedded within text files while preserving the surrounding non-structured content. ([source](https://mikefarah.gitbook.io/yq/usage/front-matter.md))

### Content Management & Publishing

- [Reference Anchors](https://awesome-repositories.com/f/content-management-publishing/content-management-systems/content-architecture-modeling/document-models/document-sectioning/anchor-links/reference-anchors.md) — The tool assigns anchor names to nodes and creates aliases to existing anchors to maintain references within structured data files. ([source](https://mikefarah.gitbook.io/yq/v3.x/commands/write-update.md))

### Programming Languages & Runtimes

- [Array-to-Map Converters](https://awesome-repositories.com/f/programming-languages-runtimes/programming-utilities/functional/array-transformation-utilities/array-to-map-converters.md) — Transforms arrays into map structures by using indices as keys. ([source](https://mikefarah.gitbook.io/yq/operators/array-to-map.md))
- [Array Element Finding](https://awesome-repositories.com/f/programming-languages-runtimes/programming-utilities/data-structure-type-helpers/data-type-utilities/array-element-finding.md) — Returns the initial item from an array or the first entry from a map, optionally filtered by expressions. ([source](https://mikefarah.gitbook.io/yq/operators/first.md))

### Scientific & Mathematical Computing

- [Temporal Arithmetic](https://awesome-repositories.com/f/scientific-mathematical-computing/numerical-mathematical-foundations/arithmetic-number-types/arithmetic-operations/temporal-arithmetic.md) — The tool adds or subtracts durations from datetime values to calculate future or past timestamps within structured data files. ([source](https://mikefarah.gitbook.io/yq/operators/datetime.md))
- [Arithmetic Operations](https://awesome-repositories.com/f/scientific-mathematical-computing/numerical-mathematical-foundations/arithmetic-number-types/arithmetic-operations.md) — The tool calculates the quotient of two numbers or splits a string into multiple segments based on a specified delimiter. ([source](https://mikefarah.gitbook.io/yq/operators/divide.md))

### User Interface & Experience

- [Expression Evaluators](https://awesome-repositories.com/f/user-interface-experience/layout-utilities/presentation-engines/template-engines/control-flow-directives/expression-evaluators.md) — Processes strings as command-line expressions at runtime to construct and execute dynamic logic. ([source](https://mikefarah.gitbook.io/yq/operators/eval.md))

### Artificial Intelligence & ML

- [Data Processing Conditionals](https://awesome-repositories.com/f/artificial-intelligence-ml/agentic-systems-frameworks/reasoning-symbolic-systems/logic-engines/logical-condition-evaluators/data-processing-conditionals.md) — The tool combines, negates, or tests sequences of values against logical criteria to determine if data matches specific requirements. ([source](https://mikefarah.gitbook.io/yq/operators/boolean-operators.md))

### Security & Cryptography

- [Data Encoding Utilities](https://awesome-repositories.com/f/security-cryptography/data-encoding-utilities.md) — Converts text content to and from Base64 format for handling encoded strings. ([source](https://mikefarah.gitbook.io/yq/usage/base64.md))

### System Administration & Monitoring

- [Data Integrity Verification](https://awesome-repositories.com/f/system-administration-monitoring/data-integrity-verification.md) — Checks files for syntax errors and reports issues to the standard error stream with non-zero exit codes. ([source](https://mikefarah.gitbook.io/yq/v3.x/commands/validate.md))