google/trillian
Trillian
Trillian is a distributed, multi-tenant verifiable data store that maintains cryptographically verifiable logs and maps using Merkle tree structures. It functions as a scalable backend for transparency logs, providing a system where data integrity is ensured through append-only records and mathematical proofs of inclusion and consistency.
The system distinguishes itself by decoupling core storage from application-specific logic through a personality layer, which handles admission criteria and data canonicalization. It employs a consensus-based leader election mechanism for high availability and a gossip protocol to synchronize views and detect split-view attacks.
Beyond its core logging capabilities, the project provides a verifiable key-value store using sparse Merkle trees and supports persistence across various relational SQL database backends. The framework includes a comprehensive security suite for managing signing keys, rotating private keys, and enforcing request quotas and rate limits per tenant.
The system is implemented in Go and provides integration tests to verify that server binaries maintain the integrity of the verifiable data store.
Features
- Verifiable Storage - Stores data in a cryptographically verifiable structure using Merkle trees to ensure transparency and integrity.
- Merkle-Tree Storage Engines - Organizes data into a cryptographic Merkle tree structure to enable efficient inclusion and consistency proofs.
- Technical Transparency Log Infrastructure - Builds systems that allow public auditors to verify the integrity and consistency of a shared set of records.
- Signed Root Distribution - Provides the latest signed root of the Merkle tree to establish the current state and size of the log.
- Cryptographic Commit Logs - Creates append-only data stores using Merkle trees to ensure records cannot be altered or deleted without detection.
- Append-Only Consistency Proofs - Creates cryptographic proofs demonstrating that a newer log version is an append-only extension of an older one.
- Distributed Consensus Stores - Provides a high-availability data store using leader election and consensus to coordinate verifiable logs.
- Log Tenant Isolation - Isolates different data streams by identifier by hosting multiple independent Merkle trees for different tenants.
- Proof-Carrying Data Stores - Runs high-availability clusters providing cryptographic proofs of inclusion and consistency for large datasets.
- Append-Only Log Feeds - Adds entries to an append-only Merkle tree using consecutive queuing or predefined ordering.
- Distributed Leader Election - Coordinates a single active master node among a distributed cluster to ensure high availability.
- Verifiable Tree Hosting - Hosts several independent Merkle trees within a single installation to support multi-tenancy.
- Cryptographic Inclusion and Consistency Proofs - Produces inclusion and consistency proofs allowing clients to verify data without downloading the entire dataset.
- Leaf Inclusion Proofs - Generates cryptographic proofs to verify that a specific entry exists within a given tree root.
- Proof Generation - Produces cryptographic Merkle proofs for specific leaf indices or hashes within the tree.
- Threshold-Signed State Management - Maintains a history of signed tree heads to establish the authoritative state and size of the log.
- Sparse Merkle Trie State Verification - Implements a key-value store using a sparse Merkle tree to provide verifiable mapping of keys to values.
- Data Canonicalization - Converts equivalent formulations of data into a single format to prevent duplication of identical entries.
- SQL-Backed Stores - Persists cryptographically hashed trees and maps across various relational database backends.
- Pluggable Database Backends - Integrates with various relational databases to persist verifiable data.
- Relational Storage Backends - Offloads the storage of tree nodes and leaf data to relational databases for durability and scalability.
- Tamper-Proof Database Managements - Records system events or financial transactions with mathematical proof of the sequence and content.
- Verifiable Map Generation - Creates cryptographically verifiable maps using a pipeline approach to scale efficiency for large datasets.
- Verifiable Map Leaf Management - Writes and verifies individual or batch leaves in a sparse Merkle tree map.
- Admission Request Validation - Validates submissions and canonicalizes data using application-specific logic to manage external interfaces.
- Gossip Protocols - Uses a gossip protocol to synchronize observer views and detect split-view attacks.
- Certificate Transparency - Maintains a verifiable log of digital certificates to prevent fraudulent issuance and improve PKI security.
- Signing Key Rotation - Updates private signing keys via an administrative interface to maintain the trust chain without altering storage.
- Inclusion Commitments - Provides a signed commitment to incorporate a submitted entry within a defined time window.
- Data Submission Validators - Validates that incoming data is signed by an authorized entity and follows a specific required structure.
- Signing Key Lifecycle Management - Handles cryptographic keys and signing operations using hardware modules or standard file formats.
- Split-View Attack Detection - Shares observer views via a gossip protocol to ensure all users see the same data.
- Verifiable Map Integrity Checks - Checks if a specific key-value pair is correctly committed within a set of map tiles.
- Application Logic Extensions - Pairs a data store with a personality layer to handle admission criteria and data canonicalization.
- Iterable and Mapping Content Validators - Applies structural and authentication checks to entries to ensure they meet format and trust requirements.
- Admission Layers - Decouples core storage mechanisms from application-specific admission criteria and data canonicalization rules.
