johnkerl/miller
Miller
Miller is a command-line data processor used for filtering, transforming, and aggregating name-indexed tabular data. It functions as a tool for querying and reshaping records across multiple file formats, serving as a converter between CSV, JSON, and YAML.
The tool distinguishes itself by using a name-indexed data model, allowing users to manipulate fields by name rather than numeric position. It utilizes single-pass streaming algorithms to compute statistics and summaries on large datasets that exceed available system memory.
Its capabilities cover data transformation and analysis, including field computation, record filtering, and data sorting. It supports the chaining of multiple operations into a linear pipeline to perform complex cleaning and statistical aggregation tasks.
Features
- Name-Indexed Data Models - Uses a name-indexed data model to allow manipulation of fields by name rather than numeric position.
- Tabular Data Processors - Provides utilities for filtering, aggregating, and joining delimited text data in CSV, JSON, and YAML formats.
- Statistical Aggregators - Computes summary statistics such as sums, averages, and counts across grouped tabular datasets.
- Data Cleaning Utilities - Filters, reshapes, and modifies name-indexed data formats using a set of command-line operations.
- Data Filtering - Retains only the records that satisfy a specific logical expression based on field values.
- Data Format Interoperability - Transforms data between formats such as CSV, JSON, and YAML to move information between systems.
- Data Format Translators - Provides a common internal representation to convert seamlessly between CSV, JSON, and YAML formats.
- Stream Processing - Processes data one record at a time to handle datasets that exceed available system memory.
- Format Converters - Transforms data between CSV, JSON, and YAML formats while preserving structured fields.
- Data Transformation Tools - Cleans and aggregates records using named fields across various formats without relying on positional indices.
- Large Dataset Streaming - Employs incremental streaming techniques to process massive files that exceed available system memory.
- Interactive Data Querying Tools - Offers a terminal-based interface for filtering and exploring structured, name-indexed data in real-time.
- Command-Line Data Processors - Serves as a CLI tool for parsing, transforming, and aggregating structured data streams.
- Command-Line Data Tools - Offers terminal-based utilities for retrieving, filtering, and displaying information from name-indexed tabular files.
- Data Operation Pipelines - Sequences multiple data transformation verbs into a linear execution flow for incremental processing.
- Computed Fields - Creates new data fields by applying mathematical or string operations to existing values.
- Data Aggregation Tools - Consolidates datasets into grouped totals or summary reports based on indexed fields.
- Streaming Aggregations - Reduces large datasets using single-pass algorithms that operate on streaming data to calculate summaries.
- Data Shaping - Removes unnecessary columns or creates new ones using programming statements to clean datasets.
- Data Sorting Engines - Provides multi-column sorting algorithms to order tabular datasets alphabetically or numerically.
- Field Selection - Extracts specific subsets of fields and reorders them for the final output.
- Field Transformations - Modifies datasets by removing unwanted columns or calculating new fields using logical expressions.
- Streaming Aggregators - Computes statistics and summaries on large datasets using memory-efficient single-pass streaming algorithms.
- Expression Evaluators - Evaluates mathematical and string expressions against record fields to generate new data columns.
- CSV and Tabular Data - Name-indexed data processing for CSV and tabular JSON.
- Command Line Utilities - Processes and queries structured data files.
