dathere/qsv

Features

• Tabular Data Wrangling - Provides a high-performance toolkit for cleaning, transforming, and reshaping tabular datasets using expressions and scripts.
• Tabular SQL Querying - Executes SQL queries and joins directly against tabular files without requiring a formal database engine.
• CSV Command Line Toolkits - Provides a high-performance suite of terminal utilities for analyzing and manipulating CSV and other delimited data files.
• Univariate Statistical Profiling - Profiles raw datasets by calculating statistics and frequency distributions to understand their structure.
• Data Profiling and Validation - Analyzes dataset schemas, calculates summary statistics, and verifies data consistency against defined rules.
• CSV Column Reorderers - Extracts, re-orders, duplicates, or drops specific columns in CSV files.
• Column Transformation - Applies string, date, and mathematical transformations to columns, including advanced detection of language and sentiment.
• Column Renamers - Provides the ability to rename columns within a dataset to improve data clarity or compatibility.
• CSV Format Converters - Modifies delimiters, quoting rules, and record terminators for ASCII delimited data.
• CSV Row Manipulators - Performs structural edits on CSV files, including dropping headers, adding enumeration columns, and exploding rows.
• CSV Row Indexes - Creates lightweight file indices to enable constant-time row counting and rapid random access slicing.
• Data Cleaning Procedures - Provides a comprehensive set of procedures for resolving encoding issues and parsing irregular date or number formats.
• Exploratory Frequency Tables - Builds tables showing the occurrence frequency of distinct values for every column in a dataset.
• External Memory Processing - Uses disk-based external memory to sort and deduplicate massive datasets that exceed available system RAM.
• Data Concatenations - Merges multiple tabular files by appending rows or joining columns.
• Data Format Translators - Translates structured tabular data between various serialization formats like CSV and JSON for tool compatibility.
• Data Quality Profilers - Analyzes tabular datasets to calculate summary statistics, frequency distributions, and infer data schemas.
• Tabular Format Bridges - Provides a unified interface to translate data between CSV, JSON, Excel, and other tabular formats.
• Regex Row Filters - Filters tabular datasets to extract rows that match specified literal or regular expression patterns.
• Dataset Joins - Executes various relational join types, including inner, outer, right, cross, anti, and semi joins.
• Multithreaded Workload Partitioning - Distributes heavy workloads across multiple CPU cores to accelerate search, join, and validation operations.
• File Format Sniffing - Infers delimiters, header rows, encoding, and mime types to correctly parse unknown tabular files.
• Flat-File SQL Analysis - Executes complex SQL queries and joins directly on flat CSV files without requiring a database engine.
• Disk-Backed Hash Tables - Eliminates duplicate records in massive datasets using disk-backed hash tables for uniqueness tracking.
• Schema Inference - Analyzes raw tabular files to automatically determine column names, data types, and structural organization.
• Descriptive Statistics Summaries - Computes detailed summary metrics, including cardinality and distribution, for each column in a dataset.
• Data Schema Validation - Verifies that tabular data conforms to JSON Schema definitions or the RFC 4180 standard.
• Tabular Duplicate Removers - Eliminates duplicate records using high-performance in-memory processing or on-disk hash tables for massive datasets.
• Large Dataset Optimizations - Implements disk-based external memory processing to sort and analyze datasets that exceed available system RAM.
• External Merge Sorting - Implements a multithreaded external merge sort to organize datasets that exceed available system memory.
• Data-to-Natural-Language Summarization - Uses OpenAI-compatible models to translate structured tabular column content into natural language summaries.
• Dataset Sampling Utilities - Provides utilities for creating reduced-size versions of large datasets using reservoir, Bernoulli, or stratified sampling.
• Multithreaded Profiling - Uses multithreaded readers to calculate row counts and frequency distributions for files exceeding system memory.
• Column-Based Partitioners - Splits a dataset into multiple separate files based on the values of a specified column.
• Column Name Sanitizers - Modifies column names to ensure compatibility with databases or data catalogs.
• Enumerated Column Generators - Inserts new columns containing incremental identifiers, UUIDs, or constant values into the dataset.
• Regex Text Replacement - Provides high-performance column content modification using regular expression search and replace patterns.
• Cross-File Consistency Checks - Performs consistency checks across multiple files and assigns confidence ratings to ensure data accuracy.
• Excel-to-CSV Converters - Extracts specific worksheets from Excel files and converts them into standard CSV format.
• CSV to JSON Conversion - Implements conversion of comma-separated values files into structured JSON formats.
• JSON to CSV Conversion - Transforms nested or line-delimited JSON structures into flat CSV tables.
• Data Ingestion - Pipes processed tabular data directly into SQL databases for downstream storage and analysis.
• Script-Based Transformations - Integrates external interpreters and scripting languages to perform complex data wrangling and custom transformations.
• Table Row Counts - Calculates the total number of records present in a tabular file.
• Dataset Slicing - Extracts specific contiguous ranges of rows from large tabular datasets.
• Schema Validators - Ensures data consistency by validating records against JSON Schema definitions and regular expression constraints.
• Row Expansion - Implements row explosion to split a single record into multiple child rows based on a column separator.
• Sort Order Verification - Checks if a dataset is correctly sorted and identifies sort breaks or duplicates.
• Symmetric-Difference Analysis - Identifies discrepancies between two comma-separated value files using high-performance structural analysis.
• Pivot Table Aggregators - Rotates data from rows to columns based on specified keys to create summary pivot tables.
• DSL-Based Wrangling - Uses a domain-specific language to create computed columns, filter rows, or aggregate data.
• Tabular File Indexing - Creates lightweight file indexes to enable constant-time row counting and rapid random access slicing of tabular data.
• Tabular Header Analysis - Displays file headers or finds the intersection of headers across multiple files.
• Date Formatting Utilities - Converts date fields into customized formats using standard strftime specifiers.
• Interactive Data Exploration Tools - Provides an interactive terminal interface for browsing and analyzing tabular data workflows in real-time.
• Multi-Format Data Exports - Exports tabular data into various structured formats including database files and spreadsheets.
• Python Scripting Engines - Integrates an external Python interpreter to process tabular data in batches for complex manipulations.
• Shell Command Execution - Loops over a dataset to trigger external shell commands for every row.
• Tabular Data Comparison - Identifies differences between two comma-separated value files using a structural grid comparison.
• Parallel Workload Distribution - Distributes data searching, joining, and validation workloads across multiple CPU cores to increase processing speed.
• Embedded Scripting Extensions - Integrates embeddable scripting to map computed columns and perform advanced data aggregations.
• Text Processing - High-performance CSV data-wrangling toolkit.

Star 历史