Open-source alternatives to OpenRefine

Apache Spark is a unified distributed data processing engine designed for large-scale data analysis and computation graphs. It functions as a distributed machine learning framework, a graph processing system, a real-time stream processor, and a SQL analytics engine. The system enables the execution of distributed SQL querying, large-scale graph analysis, and real-time stream analytics across clusters of machines. It also provides a scalable environment for implementing machine learning algorithms and predictive model development on massive datasets. The engine incorporates relational query e

Miller is a command-line data processor used for filtering, transforming, and aggregating name-indexed tabular data. It functions as a tool for querying and reshaping records across multiple file formats, serving as a converter between CSV, JSON, and YAML. The tool distinguishes itself by using a name-indexed data model, allowing users to manipulate fields by name rather than numeric position. It utilizes single-pass streaming algorithms to compute statistics and summaries on large datasets that exceed available system memory. Its capabilities cover data transformation and analysis, includin

This project is a command-line processor designed for the parsing, filtering, and transformation of structured data streams. It functions as a declarative programming environment that treats data as immutable streams, allowing users to perform complex structural modifications through the composition of small, reusable functions. By utilizing a recursive tree traversal engine, the system enables the navigation, inspection, and modification of deeply nested hierarchical data structures. The engine distinguishes itself through a stream-oriented architecture that processes input records one by on

PRQL is a functional, modular data transformation language that serves as a compiler for relational data pipelines. It allows developers to write expressive, pipelined queries that are translated into standard SQL dialects. By abstracting complex data manipulation into a readable, sequential syntax, the project enables the construction of maintainable workflows that remain independent of specific database engines. The language distinguishes itself through a robust compilation infrastructure that performs type validation and relational algebra analysis before generating target-specific code. I

Data Formulator is an automated data analysis and visualization platform that uses large language models to interpret natural language instructions for data preparation and reporting. It functions as an interactive workbench where users can clean, filter, and aggregate datasets while simultaneously generating visual representations. By combining conversational interfaces with automated transformation tools, the system enables users to explore data patterns and refine schemas without manual coding. The platform distinguishes itself through an agentic architecture that translates natural langua

Rails Admin is a web-based management dashboard and Active Record model manager for Ruby on Rails applications. It provides a graphical user interface for creating, reading, updating, and deleting database records, serving as a secure back office for database content management and administrative data auditing. The project distinguishes itself through a reflection-based schema mapping system that automatically generates CRUD interfaces from database metadata. It includes specialized tools for data versioning and change auditing to track administrative activity, as well as utilities for import

Temporal Tables is a PostgreSQL database extension and trigger function library that implements system-versioned tables and database history tracking. It automates the maintenance of historical data archives by intercepting write operations to capture previous row versions when updates and deletions occur. The extension manages record lifespans through transaction-time interval mapping, using paired timestamp columns to track active validity periods and relational schema linking to connect primary tables to their corresponding history archives. It includes concurrent conflict resolution mech

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

KnowledgeGraphCourse is a structured collection of graduate-level academic materials, lecture notes, and a comprehensive curriculum focused on the theory and application of knowledge graphs. It serves as a markdown-based educational resource that provides navigable course modules and study guides. The material covers specialized research on integrating knowledge graphs with large language models to reduce hallucinations. It includes detailed guides on using the SPARQL language for storing large-scale graph datasets and executing optimized queries. The curriculum spans a broad range of capabi

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

KeyDB is a multithreaded in-memory key-value store and distributed cache. It functions as a NoSQL database utilizing multi-version concurrency control to execute non-blocking queries and scans. The project is a multithreaded fork of Redis that maintains protocol compatibility while utilizing a multithreaded architecture to scale across multi-core hardware. It distinguishes itself with flash-tiered storage, allowing the system to offload data from primary RAM to SSD or flash storage to increase total capacity. The system supports high availability through active-active mesh replication and mu

This project is a native Windows port of the Redis in-memory key-value store. It provides a high-speed database that maintains datasets in RAM while using periodic snapshots to disk for data persistence. The implementation includes a background system service wrapper, allowing the data store to be installed as a Windows service that starts automatically upon computer boot. It utilizes native Windows compilation to enable execution directly on the host operating system. The system covers in-memory data caching and local data persistence, ensuring that information is recovered after a process

Node-cache is a lightweight in-memory data storage module designed for Node.js applications. It functions as a key-value store that maintains frequently accessed data directly within the application runtime to improve response times and reduce latency. The library manages the lifecycle of stored information through automated expiration policies, which use internal timers to remove stale entries once a defined time-to-live duration has elapsed. It provides granular control over stored resources, allowing for the manual inspection, deletion, or clearing of specific keys to manage memory usage e

Visual Insights is an automated exploratory data analysis platform and causal inference tool designed to discover patterns and cause-and-effect relationships within datasets. It functions as an interactive data visualization library using a grammar-of-graphics approach to generate multi-dimensional charts and dashboards. The project distinguishes itself through a natural language interface that translates plain-text questions into data answers and visualizations via a language model. It provides a specialized framework for causal discovery and inference, allowing users to identify variable li

DataFrame is a C++ tabular data library and manipulation engine designed for managing heterogeneous data in contiguous memory. It functions as a statistical analysis framework and time series analysis toolkit, providing the means to store, index, and transform multidimensional datasets. The project distinguishes itself through a high-performance execution model that utilizes column-major storage, SIMD-aligned memory allocation, and a thread-pool for parallel computations. It employs a visitor-based algorithm dispatch system and policy-driven transformations to decouple data processing logic f

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

Jujutsu is a distributed version control engine designed to manage project history through mutable commits and a persistent operation log. By treating the working directory as a mutable commit, it eliminates the need for manual staging areas, allowing users to modify repository history directly without checking out specific branches. The system maintains full compatibility with existing remote repositories, ensuring that local workflows remain interoperable with standard version control ecosystems. A defining characteristic of the project is its conflict-aware architecture, which treats merge

Valkey is an in-memory, NoSQL database server designed for high-performance data storage and real-time state management. It operates as a distributed key-value store, maintaining datasets entirely within system memory to facilitate sub-millisecond response times for read and write operations. The system distinguishes itself through a single-threaded event loop that utilizes asynchronous I/O multiplexing to ensure high throughput. It supports high availability via master-replica replication and provides a decoupled communication model through a built-in publish-subscribe messaging pattern. To

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

Caffeine is a high-performance caching library for the Java virtual machine designed to manage object lifecycles within the application heap. It functions as a thread-safe, memory-resident data store that reduces latency by keeping frequently accessed objects available for immediate retrieval. The library distinguishes itself through a sophisticated eviction strategy that balances recency and frequency to determine which entries to retain. It utilizes a frequency-based admission policy to evaluate the historical access patterns of new data, ensuring that the cache remains populated with the m

This project is a research data sharing framework and provenance protocol designed to ensure computational reproducibility. It provides a standardized set of guidelines for transforming raw source data into tidy formats through documented processing scripts and cleaning workflows. The framework distinguishes itself by emphasizing a strict provenance-based packaging system. It requires the organization of raw data, processing recipes, and code books into a single package, ensuring that original unmodified sources are preserved to allow for independent verification of all transformation steps.

XChange is a cryptocurrency exchange integration library and API wrapper that provides a unified interface for connecting to multiple cryptocurrency exchanges. It serves as a multi-exchange trading interface and a market data streamer, normalizing raw data from various providers into standardized data objects. The project distinguishes itself through an adapter-based normalization system and a reactive WebSocket streaming model designed to receive real-time price and volume updates while minimizing thread usage. It includes a dedicated authentication handler for managing API keys, request non

Cinder is a high-performance Python runtime implementation based on CPython. It is designed as an execution environment optimized for large-scale distributed systems and cloud environments. The project integrates a distributed memory cache and an asynchronous memory layer to manage data across multiple network nodes. It also provides a native C extension framework for developing high-performance compiled modules that link directly into the interpreter memory space. The system covers capabilities for asynchronous data retrieval, large-scale execution, and the integration of embedded scripting

Aidoku is a manga reader application and digital library manager. It serves as a modular content aggregator that allows users to discover, download, and read manga from various third-party sources and local files. The application utilizes a modular source plugin system to integrate external provider packages, enabling the ingestion of content from multiple third-party sources. It includes a sync engine that communicates with external tracking APIs to maintain consistent reading progress across different platforms. The system covers manga library management, including the ability to search fo

Flood is a BitTorrent client manager and web user interface designed to control multiple torrent client instances from a centralized dashboard. It includes a REST API for automating download management and a streaming gateway that enables the immediate playback of torrent content. The system supports multi-instance client control, allowing the administration of several separate torrent clients from one location. It features network torrent streaming, which forwards data directly to media players without requiring the full file to be written to disk. The platform provides tools for tracker op

Smallchat is a minimal implementation of a concurrent network server and TCP-based chat system designed as a demonstration of C socket programming. It provides a basic server and client architecture that enables real-time text communication between multiple connected users. The project focuses on the implementation of a networked chat server that handles simultaneous client sessions. It utilizes a centralized broadcasting model to route incoming text from one client to all other active participants. The system manages concurrent client connections and session tracking using socket descriptor

This project provides a suite of interfaces and tools for accessing electricity carbon intensity and production metrics. It includes an API for real-time and historical data, a geographic power data map for visualizing regional carbon intensity and renewable energy percentages, and a system for extracting datasets required for standardized greenhouse gas emissions reporting. The project features an interactive API sandbox that allows users to test requests and inspect data responses without writing code. It also includes mechanisms for institutional email verification to manage access to hist