medialab/xan
Xan
Xan is a command-line tool and data transformation engine for processing CSV, TSV, and JSONL datasets. It functions as a processor for compressed files, enabling random access and seeking within gzipped and Zstd files, and serves as a converter for specialized bioinformatics data formats.
The tool handles large datasets without requiring full memory loads by utilizing stream-based processing. It provides capabilities for merging, sorting, and deduplicating massive files, as well as converting data between various tabular formats.
The project covers a broad range of data wrangling and analysis tasks, including expression-based row filtering, pattern searching, and the generation of calculated columns. It includes statistical tools for computing summary metrics and frequency tables, as well as utilities for pivoting, transposing, and previewing data within a paginated terminal interface.
Features
- Command-Line Data Processors - Provides high-performance command-line utilities for manipulating, filtering, and analyzing structured CSV, TSV, and JSONL datasets.
- CSV Command Line Toolkits - A comprehensive suite of terminal-based utilities for analyzing and manipulating CSV, TSV, and JSONL datasets.
- Compressed Index Seeking - Enables fast random access and seeking within gzipped and Zstd compressed files using a separate index of offsets.
- CSV Row Manipulators - Provides comprehensive row-level operations for selecting, reordering, joining, sorting, and filtering CSV data.
- Data Format Translators - Converts structured data between different serialization formats including CSV, TSV, and JSONL.
- Large Dataset Streaming - Handles massive datasets through incremental streaming to enable merging and sorting without full memory loads.
- Format Conversions - Transforms structured data between various formats including TSV, JSONL, and specialized bioinformatics files.
- Indexed Compressed File Access - Enables fast random access and seeking within gzipped and Zstd files using a separate index of offsets.
- Indexed Compressed Readers - Reads gzipped and Zstd compressed files directly using indices for fast seeking.
- Data Transformation Pipelines - Implements a processing pipeline for calculating statistics, pivoting tables, and generating new columns via expressions.
- Row-Based Streaming - Processes data row-by-row using buffers to handle files larger than available system memory.
- Transposition Utilities - Rotates data orientation or reshapes columns into rows and vice versa.
- Biology and Bioinformatics - Handles specialized biological data files and large compressed datasets with format conversion.
- CSV Column Selectors - Extracts specific columns from a CSV file by name for focused analysis.
- Column Transformation - Modifies the contents of existing columns by applying functions or string manipulations.
- Memory-Efficient Column Selection - Loads only requested data columns into memory to reduce the resource footprint when processing wide datasets.
- Statistical Aggregators - Provides utilities to compute summary statistics like sums and averages across large datasets.
- Frequency Table Analysis - Generates tables that count occurrences of distinct values to identify data distributions within columns.
- Data Format Converters - Converts specialized bioinformatics data formats into standard tabular formats for downstream analysis.
- Sort-Merge Joins - Combines two datasets by ordering them by a common key before merging the results.
- Conditional Row Filters - Evaluates user-defined logical formulas against cell values to filter records from the dataset.
- CSV File Joiners - Combines rows from multiple CSV files using concatenation or join operations.
- Tabular Data Aggregation - Implements mathematical reduction of table columns into summary values to condense large datasets.
- Generated Columns - Generates new columns by evaluating expressions based on existing cell values.
- Text Pattern Search - Locates rows containing specific text strings or regular expression patterns within tabular datasets.
- CSV Descriptive Statistics - Calculates basic statistical metrics such as mean, variance, and standard deviation specifically for CSV columns.
- CSV Table Renderers - Formats CSV data into a readable, aligned table with paging for terminal display.
- Dataset Deduplication - Orders rows by specific columns and removes duplicate entries to ensure data uniqueness.
- Tabular Data Previews - Ships a paginated terminal interface for the rapid visual inspection and navigation of large text-based datasets.
- CSV and Tabular Data - Preview, filter, and aggregate CSV data.
- Data Formats - Tools for working with tabular data and CSV files.
- Tabular Data Processing - Command-line processor for CSV data.
- Productivity and Utilities - Data processing tool for CSV files.
