Open-source alternatives to Medialab Xan

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for

Miller is a command-line data processor used for filtering, transforming, and aggregating name-indexed tabular data. It functions as a tool for querying and reshaping records across multiple file formats, serving as a converter between CSV, JSON, and YAML. The tool distinguishes itself by using a name-indexed data model, allowing users to manipulate fields by name rather than numeric position. It utilizes single-pass streaming algorithms to compute statistics and summaries on large datasets that exceed available system memory. Its capabilities cover data transformation and analysis, includin

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

Dasel is a multi-format data processor and structured data query tool used for querying, editing, and converting data across JSON, YAML, TOML, and XML. It functions as a universal data converter and a cross-format configuration editor that allows users to interact with various serialization formats through a single unified interface. The tool provides a selector-based interface for extracting and searching nested values and modifying keys and values across different file types. It enables the transformation of data between supported structures to change file types while preserving the origina

PRQL is a functional, modular data transformation language that serves as a compiler for relational data pipelines. It allows developers to write expressive, pipelined queries that are translated into standard SQL dialects. By abstracting complex data manipulation into a readable, sequential syntax, the project enables the construction of maintainable workflows that remain independent of specific database engines. The language distinguishes itself through a robust compilation infrastructure that performs type validation and relational algebra analysis before generating target-specific code. I

omni-tools is a browser-based utility suite that provides client-side tools for manipulating PDFs, media files, and data formats. It functions as a collection of web-based processors and calculation engines that execute directly within the browser without requiring server-side processing. The suite includes a client-side PDF editor for merging, splitting, and reorganizing document structures, and a web-based media processor for resizing, trimming, and converting image and video files. It also features a data format converter that transforms structured information between JSON, CSV, and XML fo

Biopython is a bioinformatics library for Python providing tools to parse, manipulate, and analyze biological sequences, molecular structures, and phylogenetic trees. It serves as a biological sequence parser for genomic and proteomic data across multiple industry-standard file formats and acts as an interface for querying biological data and citations from NCBI Entrez repositories. The project distinguishes itself through specialized toolkits for protein structure analysis and phylogenetic tree construction. It includes a protein structure analyzer for processing PDB and mmCIF files to calcu

Data engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.

This PHP data collection library is a functional data wrapper and array manipulation framework. It converts arrays, JSON strings, and iterables into chainable collection objects designed for advanced filtering, sorting, and transformation. The library is distinguished by its ability to dynamically extend functionality through the registration of custom methods via closures. It also provides specialized capabilities for hierarchical data modeling, allowing flat datasets with parent-child identifiers to be reconstructed into nested tree structures. The toolkit covers a broad surface of data ma

This project is a comprehensive computer vision library for the PyTorch ecosystem, providing a standardized collection of neural network architectures, datasets, and high-performance transformation utilities. It serves as a foundational framework for building, training, and deploying deep learning models, offering a centralized model registry that allows developers to instantiate architectures with pre-trained weights for tasks such as image classification, object detection, and semantic segmentation. The library distinguishes itself through its modular approach to data and compute management

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

DataFrame is a C++ tabular data library and manipulation engine designed for managing heterogeneous data in contiguous memory. It functions as a statistical analysis framework and time series analysis toolkit, providing the means to store, index, and transform multidimensional datasets. The project distinguishes itself through a high-performance execution model that utilizes column-major storage, SIMD-aligned memory allocation, and a thread-pool for parallel computations. It employs a visitor-based algorithm dispatch system and policy-driven transformations to decouple data processing logic f

TextQL is a command line SQL query engine designed to execute relational queries directly against structured text files, such as CSV and TSV, without requiring a database import. It functions as a relational text file analyzer and a CSV processor that treats plain text files as virtual tables for filtering, joining, and aggregating data. The tool is built as a pipe-compatible data transformation utility, allowing it to process data from standard input and output formatted datasets. It enables relational joins across multiple files or directories within a single query to analyze relationships

This project provides a framework for performing data science tasks using command-line tools and scripts. It focuses on the processing and analysis of text and structured data directly within the terminal. The approach centers on using Unix pipes to stream data between independent processes and employing shell scripting to automate repetitive data science workflows. It utilizes plain-text interchange formats, such as CSV, to move information between diverse utilities. Capability areas include text-based data processing, command-line data analysis, and terminal-based data visualization. These

Remeda is a type-safe functional utility library for TypeScript designed for building data transformation pipelines. It provides a toolkit of helper functions for manipulating arrays and objects while maintaining strict type integrity throughout the process. The library is characterized by its support for both data-first and data-last calling styles. It utilizes lazy evaluation to process data collections, evaluating transformations only when the final result is requested to avoid creating intermediate collection copies. The toolkit covers collection manipulation, function composition, and t

sc-im is a text user interface spreadsheet calculator and data manager. It provides a keyboard-driven environment for performing mathematical computations and managing data grids within a command line interface. The application is scriptable, supporting custom functions, event-driven triggers, and the integration of external scripts to automate calculation tasks. It further allows for the loading of external compiled modules at runtime to extend its mathematical capabilities. The system covers data management through row sorting, filtering, and subtotal calculations. It supports data interop

Data Hacks is a collection of command-line utilities designed for statistical computation, real-time stream processing, and text-based data visualization. The toolkit enables users to perform rapid analysis on large datasets directly within the terminal by processing information through standard input and output streams. The project distinguishes itself through its focus on memory-efficient, stream-oriented operations that allow for the analysis of large-scale data without requiring heavy infrastructure. It utilizes stateless functional transformations and reservoir sampling to handle data st

Proselint is a prose linter and rule-based text analyzer designed to identify stylistic errors, clichés, and jargon in written text. It scans documents against a curated registry of linguistic and typographic rules to maintain professional editorial standards and improve writing quality. The project functions as a command line text processor, a programmable analysis library, and a git pre-commit hook. Its modular architecture allows the core engine to be embedded into other applications, exposed via a REST API, or integrated into text editors. The tool supports recursive directory traversal

This project is a SQL database abstraction layer that provides a consistent object-oriented interface for interacting with multiple relational database systems. It includes a driver wrapper to standardize connections and result sets, a fluent query builder for constructing portable SQL statements, and a type mapper for converting database-specific data types into native application types and vice versa. The library enables programmatic schema management through a schema manager that can introspect database metadata, model structures as objects, and generate the SQL required to migrate between

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

XlsxWriter is a library for generating spreadsheets in the XLSX format, functioning as an Excel workbook writer and file generator. It provides the capability to write data, apply cell formatting, and build complex layouts across multiple worksheets. The project distinguishes itself with a memory-optimized writing mode that flushes large datasets to disk row-by-row, enabling the creation of files exceeding 4 GB while minimizing RAM consumption. It also includes a specialized mechanism for embedding binary project files and digital signatures to enable VBA macros and signed scripts within work

This project is a Node.js client for PostgreSQL databases, providing a protocol parser to translate raw binary streams into JavaScript objects. It serves as a driver for executing queries, managing data, and integrating Node.js applications with PostgreSQL backends. The library includes a connection pool manager to reduce network overhead by caching reusable connections and a result streamer that uses cursors to retrieve large datasets incrementally. It also functions as an event listener for subscribing to asynchronous server-side notifications to trigger real-time application events. Broad

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

CUE is a constraint-based configuration language designed for data validation, schema definition, and code generation. At its core, it unifies types and values into a single concept, enabling compile-time validation that catches structural and value errors before runtime. The language treats data and constraints as the same thing, allowing a single definition to serve as both a schema and concrete configuration data. CUE distinguishes itself through its constraint-based unification engine, which combines multiple configuration sources into a single coherent result by merging their constraints

Arktype is a TypeScript runtime validation library and schema orchestrator. It synchronizes TypeScript types with runtime data validation, allowing users to define type-safe schemas that ensure unknown data adheres to specific structures during application execution. The project distinguishes itself by using set-theory type analysis to determine intersections and subtype compatibility, alongside JIT-compiled validation functions for optimized performance. It supports advanced type modeling through branded type constraints, recursive alias resolution, and the ability to generate runtime valida

geemap is a Python library and geospatial toolkit designed for interactive mapping, remote sensing visualization, and satellite imagery analysis. It serves as a Python interface for Google Earth Engine, enabling users to manage cloud-based geospatial workflows and visualize datasets within notebook environments. The project distinguishes itself through automated code conversion, allowing scripts and notebooks to be translated between programming languages. It also provides specialized remote sensing visualization capabilities, such as generating time-lapse GIFs from imagery sequences with sup

Hub is a multimodal AI data lake and vector database designed for storing and querying embeddings, text, audio, and images. It functions as a dataset version control system and a machine learning data streaming engine to support large-scale model training. The system utilizes a serverless PostgreSQL vector store to index high-dimensional embeddings for semantic search. It provides a visual interface for inspecting multimodal datasets and viewing annotations such as bounding boxes and masks. The platform handles cloud-agnostic storage synchronization and implements lazy, compressed data strea