holepunchto/hypercore
Hypercore
Hypercore is a distributed append-only logging system designed for maintaining cryptographically signed data streams that are replicated and verified across a network of peers. It provides verifiable data storage using a Merkle tree structure to ensure the integrity and authenticity of information through cryptographic proofs.
The project is distinguished by its support for sparse data replication, which allows peers to download only the specific ranges or blocks of a log required for their current needs to reduce bandwidth. It also implements encrypted peer-to-peer messaging and the ability to create read-only log snapshots at specific lengths.
The system covers a broad range of capabilities including peer-to-peer data synchronization, log integrity validation, and block-level encryption. It includes tools for managing the local storage lifecycle, such as log truncation and cache management, alongside event listeners for monitoring network connectivity and data updates.
Features
- Verifiable Storage - Maintains data in a Merkle tree structure to provide cryptographic proofs of integrity and authenticity.
- Append-Only Log Storage - Implements a distributed append-only log structure for synchronization across peer-to-peer networks.
- Sparse Replication - Supports bandwidth-efficient replication by allowing peers to download only the specific log ranges they need.
- Sparse Replication - Reduces bandwidth usage by downloading only specific ranges or individual blocks of data instead of the entire log.
- State Synchronization - Creates network streams to synchronize log state and blocks between local and remote peers.
- Peer-to-Peer Database Replication - Synchronizes log state between instances by establishing replication streams to exchange missing data blocks.
- Sparse Replication - Allows peers to fetch only required ranges of data instead of the full log to reduce bandwidth.
- Cryptographic Log Signing - Secures the log by requiring every append or truncate operation to be signed by a private key.
- Log Integrity Validation - Generates Merkle Tree hashes of the log to validate the overall authenticity and integrity of the data stream.
- Merkle Tree Utilities - Generates Merkle tree hashes and proofs to verify that specific blocks or ranges have not been tampered with.
- Merkle-Tree Data Integrity - Uses Merkle trees to provide cryptographic proofs that specific data blocks remain authentic and untampered.
- Remote Block Fetching - Enables fetching of individual data blocks by index from remote peers.
- Log Snapshots - Enables opening read-only sessions of a log at a specific length that remains frozen as data grows.
- Log Truncation - Supports shortening logs to a specific length and signing the truncation to create data forks.
- Log Content Streaming - Reads ranges of data or bytes as continuous streams supporting discrete ranges and live updates.
- Remote Data Retrieval - Provides utilities for retrieving specific blocks or ranges of bytes from remote peers.
- Range-Based Stream Queries - Supports retrieving specific segments of the log as continuous streams based on defined ranges.
- Peer-to-Peer Networking Extensions - Allows the creation of encrypted protocol streams and registration of custom extensions for peer-to-peer communication.
- Block Encryption - Applies encryption to individual blocks using specific keys to ensure the privacy of stored data.
- Message Encryption - Implements encrypted protocol streams for secure message exchange between network participants.
