deuxfleurs-org/garage

Features

• S3 Compatible APIs - Provides an S3-compatible API gateway that translates standard object storage requests into internal cluster operations.
• Cross-Location Data Replication - Distributes data copies across multiple physical locations and zones to ensure resilience against site failures.
• Distributed Data Replication - Uses conflict-free replicated data types to synchronize state and maintain consistency across distributed nodes.
• Metadata Restorations - Repairs the metadata database through synchronization from peer nodes or by restoring from snapshots.
• Data Restoration Tools - Restores missing data blocks to replacement drives by synchronizing from healthy replicas.
• CRDT Databases - Implements a distributed database architecture using conflict-free replicated data types for metadata consistency.
• Distributed Storage Recovery - Ensures service and data availability are restored after physical hardware crashes to maintain system resilience.
• Bucket Management - Provides administrative controls for creating isolated storage containers and managing their access permissions.
• Bucket Access Policies - Links API keys to specific storage buckets to restrict or grant read and write access.
• Quorum-Based Consistency - Ensures data integrity by requiring a quorum of replica acknowledgments for routing and resolving object requests.
• Cluster Node Management - Manages node identities and roles, distinguishing between storage providers and API gateways.
• Node Retirement - Excises broken nodes from the cluster and redistributes data to maintain the required replica count.
• Gateway High Availability - Employs load balancers and failover tools to ensure the S3 API gateway remains a stable entry point.
• Gateway Routing - Utilizes gateway nodes to route requests to the appropriate storage nodes for improved performance.
• Zone-Aware Data Replication - Distributes data copies across multiple physical availability zones to ensure resilience against site-wide failures.
• API Access Security - Generates unique credentials to authenticate requests and control access to specific storage buckets.
• Secure Node Networking - Secures remote procedure calls between cluster nodes using a mandatory secret handshake.
• Merkle-Tree Storage Engines - Uses Merkle trees to efficiently compare and align metadata state across distributed cluster nodes.
• Block-based State Synchronization - Audits disk storage against reference tables to identify and resolve missing data blocks.
• Missing Block Resolution - Identifies blocks that cannot be fetched from the cluster to facilitate manual retries or purging.
• Storage Deduplication - Reduces storage footprints by identifying and eliminating duplicate data blocks.
• Zone Capacity Mapping - Maps nodes to logical zones and assigns storage capacities to ensure data replication across physical locations.
• Consistency Tuning - Allows adjusting quorum requirements for read and write operations to balance consistency against availability.
• Database Metadata Snapshots - Creates consistent and simultaneous backups of metadata databases across all nodes for recovery from corruption.
• Automatic Rebalancing - Automatically redistributes data across the cluster when nodes are added or removed to maintain balance.
• Database Version Migrations - Runs migration procedures on metadata and cluster state to transition between software versions.
• Key-Value Store APIs - Provides a specialized API for bulk storage and retrieval of small metadata pairs and hash table data.
• Capacity Analysis - Reports total and available disk space for data and metadata across all cluster nodes.
• Metadata Reference Repairs - Purges orphan object versions and block references to resolve inconsistencies caused by node unavailability.
• Node Identity Replacements - Provides mechanisms to swap lost node IDs with new ones to trigger automated data rebalancing.
• Object Lifecycle Operations - Automatically expires objects or aborts incomplete multipart uploads after a specified period.
• Pluggable Database Backends - Provides a pluggable architecture allowing the use of different database backends like LMDB or SQLite for metadata.
• Pluggable Storage Engines - Supports interchangeable database backends such as LMDB, Sqlite3, and Fjall to optimize performance.
• Data Integrity - Periodically scans data directories to detect corruption and rebuilds damaged files from healthy replicas.
• CLI Administration Tools - Ships a command-line interface for performing cluster administrative tasks via an internal RPC protocol.
• Cluster Configuration Management - Includes a CLI tool specifically for defining and managing cluster-wide storage and node configurations.
• Cluster Lifecycle Managers - Provides utilities to propose, apply, or revert versioned changes to node roles and partition assignments.
• Automated Node Discovery - Implements transitive discovery using unique identifiers to allow distributed nodes to automatically join the cluster.
• Registrar-Based Node Discoveries - Integrates with orchestrators to automatically discover and connect cluster nodes without manual configuration.
• Automated Partition Balancers - Automatically redistributes data partitions across storage nodes to balance load and maintain replica counts.
• Node Availability Verification - Uses periodic TCP pings as heartbeats to verify node availability and ensure correct request routing.
• Storage Block Compression - Reduces disk space and network bandwidth usage by applying compression to data blocks.
• Latency-Aware - Improves data retrieval speed by prioritizing local nodes and low-latency zones.
• Multipart Upload Utilities - Supports chunked file transfers to allow the interruption and completion of large object uploads.
• S3 Bucket Exposures - Serves S3 API and website interfaces to clients without requiring physical data storage on the local node.
• REST Administrative APIs - Provides a dedicated REST API for programmatic management of nodes, buckets, and access credentials.
• Service Discovery Integrations - Integrates with external service discovery tools to automatically locate other nodes within the cluster.
• Scoped Token Issuance - Creates scoped, time-limited API tokens to control access to specific administrative functions.
• Client-Side Encryption - Encrypts objects using client-provided keys before writing them to disk.
• Presigned URL Generation - Creates temporary, cryptographically signed URLs that grant timed access to specific objects.
• Cluster Layout Versioning - Stages and validates versioned cluster configuration changes before committing them to the active environment.
• Rolling Updates - Implements strategies for updating binaries and configurations across the cluster incrementally to ensure continuous availability.
• API Performance Monitoring - Tracks latency, throughput, and error rates for S3 and administrative API endpoints.
• Cluster Health Monitoring - Exposes Prometheus metrics and health check endpoints to monitor distributed cluster status and quorum availability.
• Prometheus-Based Metric Exporters - Gathers operational telemetry and execution traces using Prometheus and OpenTelemetry formats.
• Replication Progress Monitoring - Tracks the progress of block resyncs and queue lengths to ensure data consistency across the cluster.
• Database Systems - S3-compatible distributed object storage for self-hosting.

Star history