jankotek/MapDB
MapDB
MapDB is an embedded database engine and disk-backed collection library that stores Java collections on disk or off-heap. It functions as a local data processing engine designed to handle datasets that exceed available physical RAM.
The project utilizes off-heap data storage to eliminate garbage collection overhead and employs disk overflow caching to balance memory and disk usage. It provides specialized utilities for filtering and analyzing large volumes of local data on a single machine.
The system ensures data integrity through ACID-compliant transactions and multi-version concurrency control. It supports persistent maps, sets, and queues using B-tree indexing and a sequential transaction log for durability.
System behavior and internal engine operations can be monitored through configurable logging levels and output formats.
Features
- Disk-Backed Data Structures - Provides implementations of standard Java data structures that persist their state directly to the local filesystem.
- ACID Transactional Cores - Implements a core transactional engine that ensures atomic and durable data modifications via a sequential log.
- Persistent Mappings - Provides disk-backed key-value mappings and other collections that persist across application restarts.
- Disk-Persistent Data Structures - Maps collections like maps, sets, and queues directly to persistent storage to survive process restarts.
- Embedded Database Engines - Functions as a database engine integrated directly into application processes without requiring external server infrastructure.
- Multi-Version Concurrency Control - Utilizes multi-version concurrency control to allow simultaneous non-blocking read and write operations.
- ACID-Compliant - Ensures atomic, consistent, isolated, and durable operations with crash safety and rollback support.
- B-Tree - Uses B-Tree structures to organize stored data for efficient disk-based range queries and lookups.
- Off-Heap Memory Managers - Manages memory outside the JVM heap to eliminate garbage collection overhead for massive datasets.
- Hybrid Memory-Disk Caches - Employs a hybrid memory-disk cache that spills less active data to disk to optimize throughput.
- Disk Caching Systems - Offloads less frequently accessed data to persistent storage to manage memory constraints and maintain performance.
- Local Data Engines - Offers an engine designed for local data storage, indexing, and high-performance processing.
- Large Dataset Optimizations - Provides specialized utilities for high-performance filtering and analysis of massive local datasets.
- Databases and Storage - Embedded database engine backed by disk or memory.
- Embedded Databases - Concurrent collections backed by disk or off-heap memory.
