Awesome GitHub RepositoriesCSV Processing

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

PapaParse is a delimited text processing library that converts CSV files into JSON objects or arrays. It provides a suite of tools for parsing delimited text and transforming structured data objects back into CSV formats through bidirectional serialization. The library is characterized by its ability to process massive datasets using incremental streaming and chunk-based processing to prevent memory overload. It includes an automatic delimiter detector to identify separator characters without manual configuration and utilizes web workers to offload parsing logic to background threads, keeping

Jackson is a Java data binding framework and multi-format data serializer used to translate data structures into native language objects. It functions as a JSON data binding library and a streaming parser that reads and writes data as discrete tokens to process large datasets with minimal memory. The project distinguishes itself through a bytecode serialization accelerator that replaces standard reflection with generated bytecode to increase data binding speed. It employs a module-based extensibility model to support a wide range of formats beyond JSON, including XML, YAML, CSV, TOML, and bin

jc is a tool that transforms plain-text results from command-line utilities, system tools, log formats, and text tables into structured JSON data. It functions as a structured data transformer capable of converting various file formats, including CSV, INI, XML, and YAML, into JSON representations for programmatic use. The project includes a collection of specific parsers for Unix commands and system tools such as df, blkid, and various package managers. It also features specialized converters for web server logs, Common Log Format, and Common Event Format strings. The tool covers broad capab

elasticsearch-dump is a command line tool for importing, exporting, and transferring data between Elasticsearch and OpenSearch instances. It functions as an index dump utility that saves documents, mappings, and analyzers to local files or standard output. The tool enables the movement of data between clusters using local files as an intermediary and can flatten nested JSON documents into CSV files for external analysis. It allows for the modification or anonymization of documents during the transfer process through the use of custom JavaScript functions. The utility covers data extraction a

Proxyman is a cross-platform HTTP debugging proxy that captures, inspects, and modifies HTTP, HTTPS, and WebSocket traffic. It functions as a man-in-the-middle proxy, decrypting SSL/TLS traffic to allow real-time inspection and modification of encrypted requests and responses. The tool is designed for debugging web and mobile applications, with capabilities for API mocking and simulation, scriptable traffic modification, and team collaboration on network logs. What distinguishes Proxyman is its deep integration with mobile and cross-platform development workflows. It provides automated certif

This project is an administrative GIS toolset that provides a comprehensive dataset of China's administrative divisions, including provinces, cities, districts, and townships. It functions as a coordinate system transformer and a boundary converter for transforming geographic data into standard formats. The toolset distinguishes itself through the ability to convert administrative boundary data between CSV, GeoJSON, Shapefiles, and SQL. It includes specialized utilities for coordinate system transformation between GCJ-02, BD-09, WGS-84, and CGCS2000 standards to ensure accuracy across differe

Webhook.site is a web-based tool that captures, inspects, and debugs incoming HTTP requests and emails sent to a unique URL, without requiring any server setup. It also functions as an API mocking and response modification tool, capable of generating mock APIs from OpenAPI specifications and altering HTTP responses, headers, and status codes for testing purposes. Beyond inspection, it serves as a platform for webhook automation and workflow orchestration, triggering multi-step automations—including database queries, SSH commands, and HTTP calls—when a webhook is received. The service distingu

csvkit is a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files directly from the terminal. It provides a suite of focused single-purpose commands that can be combined via pipes to build complex data processing workflows, with a modular architecture that includes a column-type inference engine for automatically detecting data types and a streaming-pipeline design for efficient handling of tabular data. The toolkit distinguishes itself through its SQL-engine abstraction layer, which allows users to run SQL queries directly against CSV files without req

GAM is a command-line tool for administering Google Workspace and Cloud Identity. It translates command-line arguments into structured API calls, enabling administrators to manage users, groups, organizational units, and domain settings across a Google Workspace environment. The tool handles authentication through OAuth2 flows, service accounts, and workload identity federation, and supports multi-tenant configurations for managing multiple domains or cloud projects from a single installation. GAM distinguishes itself through its batch processing and automation capabilities. It can process la

Mapshaper هي أداة لمعالجة وتبسيط وتحويل البيانات المتجهة الجغرافية، متاحة كواجهة سطر أوامر، وأداة متصفح ويب، ومكتبة Node.js. تعمل كمسقط للإحداثيات، ومحول للبيانات المتجهة، ومحسن لأصول خرائط الويب مصمم لتحويل مجموعات البيانات المكانية بين أنظمة مرجعية إحداثية وتنسيقات ملفات مختلفة. يتميز المشروع بتبسيط الهندسة مع الحفاظ على الطوبولوجيا، مما يقلل من عدد الرؤوس مع الحفاظ على الحدود المشتركة لمنع الفجوات والتداخلات. كما يعمل على تحسين الأصول للويب من خلال تكميم الإحداثيات وتصفية السمات لتقليل أحجام الملفات. يغطي النظام مجموعة واسعة من الإمكانيات، بما في ذلك إعادة إسقاط الإحداثيات باستخدام سلاسل PROJ ورموز EPSG، وتحويل البيانات عبر تنسيقات مثل Shapefile وGeoJSON وTopoJSON وGeoPackage وKML. ويوفر أدوات معالجة هندسية واسعة النطاق للتخزين المؤقت، والقص، والإذابة، وإصلاح الطوبولوجيا، بالإضافة إلى أدوات إدارة البيانات لربط السمات وتصفيتها وتحويلها. بالإضافة إلى ذلك، يتضمن ميزات تصور لتوليد صادرات SVG مصممة، وشبكات إحداثيات، وخرائط رموز متناسبة. يمكن دمج إمكانيات المعالجة المكانية مباشرة في تطبيقات JavaScript وخطوط أنابيب البناء عبر مكتبة Node.js الخاصة به.

Exportify is a web-based music metadata tool and data archive utility designed to export Spotify playlists and liked songs into portable CSV files. It serves as a multi-language data export utility that allows users to save track, album, and artist details for long-term preservation or data analysis. The tool provides capabilities for bulk playlist archiving, where entire collections of playlists are saved as CSV files and bundled into a single ZIP archive. It also supports targeted extraction through playlist search and filtering based on ownership or collaboration status. The system handle

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for

Tippecanoe is a command-line tool used to generate optimized vector tiles for web maps. It converts large-scale geospatial datasets, including GeoJSON, CSV, and Geobuf files, into binary vector tiles or MBTiles SQLite databases. The project is designed to maintain map performance and visual quality across different zoom levels. It achieves this through geospatial data downsampling, which includes simplifying geometries and thinning point density to prevent tile overcrowding and keep tile sizes within specific limits. The tool provides extensive data transformation capabilities, such as attri

This project is an Amazon web scraper and e-commerce data extractor designed to retrieve product names, prices, and ratings. It functions as a headless browser crawler that converts unstructured web content from product listings into structured JSON and CSV formats. The tool incorporates anti-bot bypass capabilities to circumvent CAPTCHAs and security challenges. It achieves this through the use of residential proxy integration, automatic proxy rotation, and the modification of browser fingerprints to simulate human interaction patterns. The system provides broad web scraping capabilities, i