Backup, disaster recovery and chaos

Redis is an in-memory, key-value database designed to provide sub-millisecond latency for read and write operations. It functions as a versatile data platform, serving as a distributed cache, a message broker, a NoSQL document store, and a vector database. The system utilizes an event-driven, single-threaded loop to process requests efficiently, while maintaining data durability through append-only persistence logs and asynchronous snapshotting mechanisms. What distinguishes Redis is its ability to handle complex data structures—including strings, hashes, lists, sets, and sorted sets—alongside hierarchical JSON documents and high-dimensional vector embeddings. It supports advanced operational patterns such as active-active database deployment for global distribution, real-time data streaming, and probabilistic statistics for large-scale data analysis. These capabilities are complemented by a pluggable indexing engine that enables semantic similarity matching and full-text retrieval. The platform offers a comprehensive ecosystem for managing distributed state, including master-replica replication, automated cluster management, and granular security controls like access control lists and TLS encryption. Developers can interact with the database through language-specific client libraries that support connection multiplexing and object mapping, or via a command-line interface for direct administrative tasks and scripting. Redis is deployed through standard package managers and supports both self-managed clusters and managed cloud instances. Observability is provided through integrated tools for performance analysis, slow log monitoring, and bulk data management.

TiDB is a horizontally scalable, distributed SQL database designed to provide consistent transactional storage and high-performance analytical processing within a single unified architecture. It utilizes a decoupled compute-storage design and a distributed key-value storage layer to ensure horizontal scalability and efficient range-based queries. By employing a consensus-based replication algorithm, the system maintains high availability and automatic failover across multiple nodes and geographical regions. The platform distinguishes itself through its hybrid transactional and analytical processing capabilities, which allow complex SQL queries to run against replicated columnar data without disrupting primary transactional workloads. It also integrates high-dimensional vector search functionality, enabling semantic similarity queries directly alongside traditional relational data. To support diverse operational needs, the system provides native tools for real-time data streaming, seamless migration from external database systems, and multi-region disaster recovery. The database is built for cloud-native environments, offering comprehensive lifecycle management through Kubernetes operators that automate deployment, scaling, and rolling upgrades. It maintains compatibility with standard SQL interfaces, allowing applications to connect using common drivers while managing complex concurrency through pessimistic transaction handling. Detailed documentation and command-line utilities are available to assist with cluster orchestration, performance troubleshooting, and the configuration of production-grade topologies.

Immich is a self-hosted media management platform designed to provide a centralized, private repository for photos and videos. It functions as a comprehensive system for organizing, backing up, and viewing personal media collections across mobile devices, web browsers, and external storage locations. By maintaining full control over data ownership and storage infrastructure, the platform ensures that users retain sovereignty over their digital assets. The system distinguishes itself through a distributed architecture that coordinates background media synchronization, real-time filesystem monitoring, and automated deduplication. It leverages an integrated machine learning pipeline to perform intelligent asset organization, including facial recognition, object detection, and metadata extraction. These processes are executed through containerized service orchestration, which manages complex dependencies and hardware-accelerated tasks within isolated environments. Beyond core management, the platform provides extensive tools for disaster recovery and library maintenance. Users can configure automated database backups, manage external storage volumes, and define granular synchronization policies for mobile devices. The system also includes command-line utilities for secure remote operations, such as authenticated asset uploading and server version verification, ensuring compatibility and consistency across distributed deployments.

etcd is a distributed, strongly consistent key-value store designed to provide reliable storage for critical system metadata and coordination primitives. It functions as a distributed consensus engine, utilizing a replicated log and leader-based state machine to ensure that all nodes in a cluster maintain a synchronized view of data. By providing atomic operations and linearizable reads and writes, it serves as a foundational component for distributed systems requiring high availability and fault tolerance. The system distinguishes itself through its multi-version concurrency control, which enables non-blocking read operations while maintaining strict consistency for concurrent writes. It supports complex distributed coordination through features like lease-based expiration, which allows for the automatic removal of data based on client activity, and asynchronous key change monitoring, which provides real-time event notifications for data modifications. These capabilities are supported by a persistent B-tree-based storage engine and write-ahead logging to ensure durability across system crashes. Beyond its core storage functions, the project provides a comprehensive suite of tools for cluster management, including automated peer discovery via DNS or service registries and robust security enforcement. It includes built-in mechanisms for transport layer security, role-based access control, and certificate management to protect data in transit and at rest. Operational reliability is further maintained through snapshot-based disaster recovery, cluster health monitoring, and granular performance tuning for disk and network resources. The system is configured through structured files or command-line flags, allowing for flexible deployment across diverse infrastructure environments.

This project is a comprehensive educational resource focused on the principles, patterns, and trade-offs required to design scalable, reliable, and high-performance distributed systems. It provides a structured curriculum that covers the fundamental architectural strategies necessary for building modern software infrastructure, ranging from high-level system decomposition to low-level networking and data management. The repository distinguishes itself by offering deep dives into complex architectural patterns, such as microservices-based decomposition, event-driven communication, and command-query responsibility segregation. It provides detailed comparisons of API design techniques, including REST, GraphQL, and gRPC, while offering guidance on when to utilize specific patterns like the backend-for-frontend approach or circuit breakers to manage service failures and maintain system stability. Beyond core architecture, the project explores a broad capability surface including infrastructure planning, database sharding, caching strategies, and security standards like OAuth and OpenID Connect. It also addresses operational reliability through service discovery, rate limiting, and disaster recovery planning, providing a technical reference library designed to assist engineers in navigating complex design discussions and technical interviews.

TDengine is a distributed time-series database designed for the high-speed ingestion, compression, and retrieval of timestamped metrics and sensor data. It functions as a SQL-compatible analytics engine, allowing users to perform complex operations on massive volumes of time-ordered information using standard relational syntax. The platform is built to serve as a backend foundation for industrial IoT environments, managing real-time data streams and device metadata through a cluster-based architecture. The system distinguishes itself through a distributed sharding architecture that uses consistent hashing to ensure horizontal scalability and high-throughput ingestion. It employs a log-structured write path to minimize disk seek latency and utilizes super-table virtualization to provide a unified logical view across multiple physical tables. To maintain performance and cost-efficiency, the database features automated multi-tiered lifecycle management, which migrates data between high-performance memory and low-cost storage based on age and access frequency. Beyond its core storage capabilities, the platform provides robust tools for edge-to-cloud synchronization, ensuring consistent data states across geographically distributed infrastructure. It includes built-in support for real-time stream processing, allowing for the analysis of live data without requiring external message queues. The system also incorporates comprehensive security frameworks, including user access control, audit logging, and encrypted transport protocols to protect sensitive operational data. Developers can interact with the database through native client libraries that support connection pooling and query parameter binding. The system is documented with comprehensive error code diagnostics and provides command-line utilities for cluster administration, health monitoring, and configuration management.

This project is a command-line utility designed for secure, content-addressable data archiving. It functions as an encrypted backup tool that stores data as deduplicated chunks, ensuring that every piece of information is identified by a cryptographic hash to maintain integrity across all backups. By applying strong encryption and message authentication codes to both data and metadata, the software prevents unauthorized access and detects potential tampering. The tool distinguishes itself through a backend-agnostic storage abstraction that allows users to maintain repositories across diverse environments, including local filesystems, network-attached storage, and various cloud object storage providers. It optimizes storage efficiency and network performance by aggregating small data chunks into structured pack files and utilizing index-based metadata lookups. To further improve performance, the system maintains a local cache of repository indexes, which accelerates search operations and reduces latency during backup analysis. Beyond its core storage capabilities, the software supports automated backup orchestration and disaster recovery planning through versioned snapshots. It provides a comprehensive set of management tools for inspecting repository objects and configuring secure connections to remote backends via standard protocols. The software is distributed as a portable binary, with support for installation through native package managers, containerized execution, and cross-compilation from source.

bup is a deduplicating backup manager and incremental backup system. It uses a Git packfile-based storage format to eliminate redundant data across files and versions, treating every incremental save as a full backup. The system provides secure remote transport interfaces for transferring and managing backup data on remote servers via SSH. It also includes a backup repository browser available as both a web interface and a filesystem mount for exploring and retrieving files from snapshots. The project covers broad capability areas including disaster recovery, repository administration, and storage optimization. It utilizes rolling checksum chunking for data reduction, garbage collection for space reclamation, and integrity auditing to validate backup archives. Recovery features support target version restoration, sparse file handling, and the restoration of POSIX1e ACLs.

Litestream is a database backup utility that provides continuous, incremental replication for SQLite databases. It operates as a background process that monitors local database files and streams modifications to remote cloud storage, ensuring that off-site backups are maintained without manual intervention. The tool functions by intercepting the database file system layer to capture page-level changes and tailing the write-ahead log. This approach allows for real-time synchronization of transactions to various cloud object storage providers through a unified abstraction layer. Beyond continuous replication, the system supports disaster recovery by enabling the reconstruction of database states. It utilizes periodic compressed snapshots combined with archived log files to facilitate point-in-time recovery, allowing users to restore a database to a specific historical moment.

Incus is a unified orchestration platform for managing system containers, OCI application containers, and virtual machines through a single control plane. It brings together cluster infrastructure management, secure multi-tenancy, software-defined networking, and pluggable storage backend orchestration into one cohesive system exposed via a full REST API and command-line interface. What distinguishes Incus is its ability to run multiple instance types side by side—full Linux system containers, OCI application containers, and QEMU virtual machines—all managed with consistent tooling. Networking is handled through OVN-based virtual networks with built-in ACLs and BGP route advertisement, while storage uses a driver abstraction layer that supports Btrfs, ZFS, LVM, Ceph, LINSTOR, and directory backends. Clustering is built on Raft consensus for high availability, and containers use user-namespace isolation with non-overlapping UID/GID maps to prevent privilege escalation. Authentication supports TLS client certificates, OpenID Connect, PKI, and ACME certificate issuance, with fine-grained authorization via role-based access control and OpenFGA integration. The platform also provides comprehensive image management, backup and recovery workflows, real-time monitoring and metrics export to Prometheus and Grafana, and integration with infrastructure-as-code tools such as Terraform and Ansible. Cluster operations include automatic rebalancing, live migration, and rolling upgrades.

MinIO is a software-defined, cloud-native object storage server designed to manage large volumes of unstructured data. It functions as a distributed storage cluster that aggregates multiple independent nodes into a unified, scalable pool, providing a high-performance infrastructure compatible with standard cloud storage protocols and application programming interfaces. The system utilizes a shared-nothing architecture that eliminates central metadata servers, relying instead on a decentralized hash table to map objects across the cluster. Data availability and resilience are maintained through erasure coding, which distributes data fragments across multiple drives to protect against hardware failure. To ensure long-term data integrity, the system performs continuous background scanning to detect and repair silent corruption. It also supports multi-tenant environments by providing logical isolation for buckets and user credentials, allowing for secure, self-hosted data management across private or hybrid cloud deployments. Beyond its production capabilities, the software provides a consistent environment for local development and testing of data-intensive applications. Administrative tasks, cluster monitoring, and data operations are managed through a unified command-line client or an embedded web-based browser. The software can be deployed by building container images or by compiling the source code directly.

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It employs a language-agnostic intermediate representation to synthesize these definitions into platform-specific configurations, while supporting aspect-oriented policy injection to apply security and compliance rules across infrastructure definitions during the synthesis phase. Beyond core provisioning, the project provides a modular component registry for distributing and reusing pre-configured infrastructure building blocks. It supports multi-account orchestration, allowing for the deployment of consistent resource sets across different regions and accounts from a single template, and includes capabilities for detecting infrastructure drift to ensure deployed environments remain aligned with their defined state. The project is distributed as a software development kit, providing programmatic interfaces to manage the full lifecycle of cloud resources and integrate infrastructure definitions directly into application codebases.

SeaweedFS is a distributed object store and high-performance file system designed to manage massive volumes of unstructured data. It utilizes a decoupled architecture that separates metadata management from raw data storage, allowing for independent scalability and the efficient handling of billions of files. By providing a POSIX-compliant interface, it enables applications to interact with a unified namespace while maintaining the performance characteristics of a distributed object store. The system distinguishes itself through a multi-region data fabric that supports active-active replication across geographically distributed clusters, ensuring high availability and disaster recovery. It employs a direct-access data path that allows clients to communicate with volume servers without involving the master server, maximizing throughput. Furthermore, the platform integrates erasure coding for fault tolerance and automated tiered storage lifecycle policies to optimize data placement based on access frequency. Beyond its core storage capabilities, the project functions as a cloud-native orchestrator for data lakes and content delivery infrastructure. It supports multi-cloud synchronization and provides secure administrative access through standard authentication protocols. The software is available as binary or container images for deployment across heterogeneous infrastructure.

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network outages or service restarts. Its schema-agnostic processing model allows for dynamic field manipulation and enrichment, enabling users to normalize telemetry data from disparate sources without requiring rigid, predefined schemas. The platform supports a wide range of deployment topologies, operating as a lightweight edge agent on individual hosts or as a centralized aggregator for high-volume data processing. It provides extensive integration capabilities for cloud-native environments, including automated log collection from containers and native support for various cloud storage and monitoring services. Vector is configured via a declarative engine that validates pipeline definitions and supports dynamic reloads without service interruptions. The software is distributed as a pre-compiled binary and can be installed via standard system package managers or containerized deployment methods.

Cockroach is a distributed SQL database designed to scale horizontally across multiple nodes while maintaining strict ACID compliance and global data consistency. It functions as a relational database engine that automatically partitions data into ranges, rebalancing them across a cluster to accommodate growing storage and throughput requirements. By utilizing a distributed consensus protocol, the system ensures that all nodes agree on the order of operations, providing fault tolerance and continuous availability even in the event of hardware failures. The system distinguishes itself through a layered architecture that separates the relational SQL abstraction from a distributed key-value store. It achieves global consistency without requiring perfectly synchronized hardware clocks by employing a hybrid logical clock synchronization mechanism. To support high-concurrency environments, it utilizes multi-version concurrency control and lock-free transaction execution, which allow for consistent snapshots and efficient conflict resolution. Furthermore, the engine is built for compatibility, implementing the standard wire protocol to support existing relational database drivers and tools. Beyond its core transactional capabilities, the platform includes comprehensive tooling for cluster orchestration, security, and performance diagnostics. It supports a variety of deployment models, ranging from self-hosted on-premises configurations to fully managed cloud services. The system provides a command-line interface for session management and query execution, ensuring that administrators can monitor cluster health and manage workloads through standard relational interfaces.

Kubeasz is an automation framework designed for the lifecycle management of production-grade Kubernetes clusters. It functions as an Ansible-based provisioner that orchestrates the installation, scaling, and maintenance of cluster components across distributed Linux nodes. By utilizing inventory-driven management and role-based task modularization, the project ensures that infrastructure configurations remain consistent and reproducible across diverse environments. The platform distinguishes itself through its focus on automated system administration and operational continuity. It provides built-in capabilities for performing version upgrades and rotating security certificates without interrupting active services. Furthermore, the tool integrates disaster recovery workflows, allowing administrators to create snapshots of the cluster state and restore the entire environment to a functional condition following data loss or system corruption. Beyond core lifecycle operations, the project covers a broad range of infrastructure tasks including network traffic routing and load balancing configuration. It employs template-based generation and idempotent state reconciliation to manage service settings and ensure that target nodes align with defined infrastructure requirements. The project is distributed as a collection of Ansible playbooks, providing a structured approach to managing the full operational lifecycle of a Kubernetes cluster.

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven tool access, which enforce security boundaries by restricting agent operations based on defined functional roles. It utilizes context-aware task routing to match incoming requests with specific agent capabilities and model performance profiles, while implementing deterministic fallback mechanisms to maintain operational continuity when agents encounter errors or context limits. This architecture allows for modular capability expansion and reproducible environment configurations through version-controlled templates. The system covers a broad capability surface, including automated technical documentation, cloud infrastructure management, and security auditing. It supports diverse domains such as API design, database optimization, and system reliability engineering, providing tools for incident response, performance monitoring, and compliance enforcement. These capabilities are integrated into a command-line interface that enables developers to search, fetch, and deploy specialized subagents directly from the repository.

Syncthing is a decentralized file synchronization engine that maintains consistent data states across multiple devices through peer-to-peer mesh networking. It operates as a background daemon that automatically replicates file creations, modifications, and deletions between trusted nodes without requiring central servers. By utilizing content-addressable block indexing and block-level delta synchronization, the system identifies and transfers only the modified segments of files, ensuring efficient data propagation across heterogeneous environments. The project distinguishes itself through a security-first architecture that relies on mutual TLS authentication to verify device identity, ensuring that all connections are cryptographically bound to trusted certificate fingerprints. It supports flexible synchronization modes, including bidirectional replication, unidirectional mirroring for backups, and reference-based enforcement. For added privacy, the system provides folder-level encryption for untrusted devices and allows for granular control over network traffic, including the ability to restrict operations to local networks or utilize relay infrastructure for NAT traversal. Beyond its core replication capabilities, the platform offers comprehensive management tools, including a web-based dashboard for monitoring connection status and throughput, as well as a command-line interface for advanced configuration. It includes robust versioning strategies to protect against data loss and supports complex deployment scenarios through native service integration and observability metrics. The software is designed for cross-platform compatibility and can be installed via standard package managers or containerized environments.

Mattermost is a self-hosted, enterprise-grade communication platform designed for organizations that require strict control over their internal data and messaging infrastructure. It functions as a centralized hub for real-time team interaction, offering persistent messaging, voice and video conferencing, and integrated project management tools within a single, private workspace. The platform is built to support high-security environments, including air-gapped deployments where public internet access is restricted or unavailable. The platform distinguishes itself through a focus on regulatory compliance and administrative sovereignty. It provides granular role-based access control, comprehensive audit logging, and data retention policies to meet legal and security standards. Organizations can extend the core functionality through a plugin-based framework, allowing for the injection of custom server-side logic and UI components without modifying the underlying source code. Furthermore, the system acts as a secure workflow orchestrator, enabling teams to integrate automated tasks and external services directly into their communication channels. The architecture is designed for scalability and reliability, supporting large-scale deployments through Kubernetes-based orchestration and microservices-ready infrastructure. Administrators can manage complex environments using centralized identity federation, external search indexing for high-performance data retrieval, and robust disaster recovery planning. The platform also includes tools for mobile device management and custom branding to ensure a consistent and secure experience across organizational hardware. Comprehensive documentation is available to guide administrators through installation, configuration, and maintenance, including specific procedures for Kubernetes deployments and air-gapped environment setups.

Kubernetes is a distributed container orchestration platform that automates the deployment, scaling, and management of containerized applications across clusters of computing nodes. It functions as a declarative infrastructure controller, utilizing a control loop architecture that continuously monitors the current system state against user-defined configurations to ensure desired operational outcomes. The system relies on a centralized API-driven interface and a replicated key-value store to maintain a consistent source of truth for all cluster objects. The platform distinguishes itself through a highly extensible design that allows users to define domain-specific objects using the same native API and control loop infrastructure. It employs a standardized abstraction layer for container runtimes, enabling modular execution engines, and utilizes a pluggable controller pattern that supports third-party integrations without requiring modifications to the core codebase. An algorithmic bin-packing engine further optimizes hardware utilization by dynamically matching workload requirements with available cluster capacity. Beyond core orchestration, the system provides comprehensive operational support for distributed environments, including automated lifecycle management, horizontal and vertical scaling, and self-healing mechanisms that maintain service availability. It encompasses integrated solutions for networking, persistent storage orchestration, and secure secret management. Diagnostic utilities for monitoring performance metrics, aggregating logs, and troubleshooting infrastructure-level issues are also included to support cluster health and reliability.