wireservice/csvkit
Csvkit
csvkit is a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files directly from the terminal. It provides a suite of focused single-purpose commands that can be combined via pipes to build complex data processing workflows, with a modular architecture that includes a column-type inference engine for automatically detecting data types and a streaming-pipeline design for efficient handling of tabular data.
The toolkit distinguishes itself through its SQL-engine abstraction layer, which allows users to run SQL queries directly against CSV files without requiring a database server, treating them as database tables for flexible analysis. It also offers a format-agnostic serialization bridge for converting between CSV, JSON, Excel, and fixed-width formats, along with an in-memory aggregation engine for computing summary statistics and an interactive Python shell that pre-loads CSV data as lists for ad-hoc analysis.
Beyond its core identity, csvkit covers a broad range of CSV data operations including inspection of file structure and schema, cleaning and validation to remove duplicates and fix malformed rows, filtering and sorting by column values, joining multiple files on common columns, and splitting data based on column values. It also supports database integration for importing CSV data into PostgreSQL and exporting query results back to CSV, as well as formatted terminal display of tabular data as aligned tables.
Features
- CSV Command Line Toolkits - Provides a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files.
- Inspectors and Cleaners - Inspects CSV structure, detects schema, removes duplicates, and cleans malformed rows for data preparation.
- CSV - Runs SQL queries directly against CSV files, treating them as database tables for flexible analysis.
- CSV Column Reorderers - Reorders columns in CSV files from the command line.
- CSV Column Reorderers - Selects and reorders columns in CSV files from the command line.
- CSV Column Selectors - Selects columns from CSV files by name from the command line.
- Excel-to-CSV Converters - Converts Excel spreadsheets into CSV format using a single command-line invocation.
- CSV to JSON Conversion - Outputs CSV data as JSON, enabling interchange with web and application formats.
- JSON to CSV Conversion - Transforms JSON data into CSV format through a straightforward command-line operation.
- CSV Record Filters - Applies SQL-like filtering, sorting, and aggregation to CSV files from the command line without a database.
- CSV Row Manipulators - Provides comprehensive row and column manipulation for CSV files from the command line.
- CSV Row Transformers - Filters, sorts, and transforms rows and columns in CSV files from the command line.
- CSV Row Sorters - Sorts rows in CSV files by column values from the command line.
- CSV Row Sorters - Sorts rows in CSV files by column values from the command line.
- CSV File Joiners - Joins CSV files on common columns using command-line operations.
- CSV File Splitters - Splits a CSV file into multiple files based on distinct column values.
- Data Format Converters - Converts CSV data to and from JSON, Excel, and SQL database formats via single commands.
- CSV File Joiners - Joins CSV files on common columns using command-line operations.
- CSV Column Selectors - Selects and reorders columns in CSV files by name or index from the terminal.
- Tabular Format Bridges - Converts between CSV, JSON, Excel, and fixed-width formats through a unified serialization bridge.
- CSV Query Engines - Runs SQL queries directly against CSV files, treating them as database tables for flexible analysis.
- CSV File Joiners - Joins CSV files on common columns using command-line operations.
- SQL Engines - Translates SQL queries into in-memory operations on CSV data without requiring a database server.
- SQL Query Builders - Runs SQL queries directly against CSV files, treating them as database tables for flexible analysis.
- CSV - Runs SQL queries directly against CSV files to filter, join, and aggregate tabular data.
- Format Conversions - Converts between CSV and formats like Excel, JSON, SQL, and fixed-width files using a single command.
- Format-Agnostic Converters - Converts between CSV and other tabular formats such as Excel, JSON, and SQL, preserving column types and structure.
- CSV File Stackers - Stacks multiple CSV files with identical columns into one combined file.
- Command Line Tooling - Provides a suite of composable single-purpose CLI commands that pipe together for CSV workflows.
- CSV File Joiners - Joins CSV files on common columns using command-line operations.
- CSV Row Manipulators - Provides comprehensive row and column manipulation for CSV files from the command line.
- CSV Descriptive Statistics - Computes summary statistics like count, mean, min, max, and standard deviation for CSV columns.
- Tabular Data Type Inference - Automatically detects column data types (numeric, date, string) by sampling CSV values.
- PostgreSQL Bulk Importers - Imports CSV data into PostgreSQL tables using a command-line insert flag.
- SQL-to-CSV Exporters - Exports SQL query results from databases directly to CSV files.
- Data Cleaning Procedures - Applies operations like removing duplicates, filling missing values, and reformatting columns from the terminal.
- CSV Import Managers - Loads CSV data directly into PostgreSQL databases, creating tables as needed.
- Data Import and Export - Exports PostgreSQL query results and imports CSV data into databases using command-line tools.
- CSV Exports - Exports database query results to CSV format using a single command-line invocation.
- Conditional Row Filters - Selects rows from CSV files that match numeric comparisons or text matches via command-line expressions.
- Keyword Row Filtering - Searches CSV rows for cells matching a regular expression pattern and outputs matching rows.
- Regex Row Filters - Finds rows in CSV files where a specified column matches a given regular expression.
- CSV - Displays column names, data types, and summary statistics of a CSV file to understand its contents.
- Schema Inspectors - Displays column names, data types, and sample values to help understand a CSV file's structure.
- CSV Exporters - Exports PostgreSQL query results directly to CSV files without a separate database client.
- In-Memory Execution Engines - Computes summary statistics and aggregations entirely in memory using Python data structures.
- Structured Data Inspections - Displays column names, data types, and sample values from a CSV file to understand its schema.
- CSV Object Mappings - Maps CSV rows to Python lists and columns to typed objects for direct manipulation.
- Pipeline and Processing Architectures - Processes CSV data through a chain of Unix-style commands where each tool reads from stdin and writes to stdout.
- CSV Data Visualizers - Displays column names, data types, and summary statistics of a CSV file to understand its contents.
- CSV Table Renderers - Renders CSV data as formatted tables in the terminal.
- CSV and Tabular Data - Suite of tools for converting and analyzing CSV files.
- Data Analysis and Visualization - Command-line tools for converting and working with CSV data.
- Data Loading Extraction - Utilities for converting and working with CSV.
- Document and File Processing - Offers command-line utilities for working with CSV data.
- Command Line Utilities - Provides tools for manipulating CSV data.
