microsoft/FASTER
FASTER
FASTER is a high-throughput key-value store that combines an in-memory data store with a hybrid memory-disk storage engine, enabling datasets larger than available RAM. It uses a latch-free, cache-optimized index for concurrent point lookups and heavy updates, and records all mutations to a persistent append-only log on disk with checksum validation and group-commit checkpointing for crash recovery.
The system supports multi-key transactional workloads through atomic multi-key locking, ensuring transactional consistency without coarse-grained contention. It exposes the key-value store to remote clients over a custom TCP protocol that scales linearly with the number of concurrent connections, and provides atomic value merging for read-modify-write operations.
FASTER includes a non-blocking checkpoint-recovery model that restores consistent state after crashes, and provides high-performance iterators for reading through the persistent log sequentially. The append-only log engine supports frequent low-latency commits and saturates disk bandwidth, while the hybrid storage layer seamlessly spills cold data to fast local or cloud storage.
Features
- Key-Value Stores - Provides a high-throughput key-value store with latch-free indexing and hybrid memory-disk storage.
- Key-Value - Provides a high-throughput key-value store with concurrent operations for point lookups and heavy updates.
- Latch-Free Indexes - Uses a latch-free, cache-optimized index for concurrent point lookups and heavy updates.
- Disk-Backed Stores - Manages a key-value store larger than RAM by spilling cold data to fast external storage.
- Append-Only Log Stores - Records all mutations to a persistent append-only log with checksum validation and group-commit checkpointing.
- Memory-Disk Layering - Keeps hot data in memory and seamlessly spills cold data to fast local or cloud storage.
- Hot-Cold Data Spilling - Implements a hybrid storage engine that seamlessly spills cold data to fast local or cloud storage.
- Write-Ahead Logging - Logs operations to a high-performance concurrent persistent log that saturates disk bandwidth.
- Checkpoint-Based Recovery - Restores consistent key-value store state after crashes using non-blocking checkpointing.
- Custom Protocol Servers - Implements a custom TCP protocol for remote key-value store access that scales linearly with connections.
- Custom Protocol Servers - Exposes the key-value store over a custom TCP protocol that scales linearly with concurrent connections.
- Custom Protocol Access - Exposes the key-value store to remote clients over a custom TCP protocol.
- Multi-Key Locking Mechanisms - Provides atomic multi-key locking for transactional consistency without coarse-grained contention.
- Group-Commit Checkpoints - Restores consistent state from checkpoints using a non-blocking group-commit recovery model.
- Group-Commit Checkpoints - Restores consistent state after crashes using non-blocking group-commit checkpointing.
- Custom Value Merging - Provides atomic value merging for read-modify-write operations with user-defined merge functions.
- Atomic Multi-Key Locking - Provides a locking subsystem that atomically locks multiple keys for transactional consistency.
- Checkpoint-Based Recovery Logs - Logs operations to a persistent log that supports crash recovery from checkpoints.
- Caching - High-performance key-value store for large datasets.
- Database Systems - High-performance persistent key-value store.
