1 repository
Processes for ensuring all records maintain a consistent length via padding or truncation.
Distinct from Record Transformers: Distinct from structural transformation by specifically focusing on field length consistency across records.
Explore 1 awesome GitHub repository matching data & databases · Length Normalization. Refine with filters or upvote what's useful.
qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for
Forces all records to have the same length by padding or truncating fields.