# Vonng/ddia

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/vonng-ddia).**

22,648 stars · 4,497 forks · Python · cc-by-4.0

## Links

- GitHub: https://github.com/Vonng/ddia
- Homepage: https://ddia.vonng.com
- awesome-repositories: https://awesome-repositories.com/repository/vonng-ddia.md

## Topics

`book` `database` `ddia` `distributed-systems`

## Description

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure.

The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, while also examining the architectural patterns for both batch and stream processing pipelines.

Beyond foundational theory, the project covers the implementation of event-driven systems, including event sourcing, log-structured storage, and message brokering. It addresses the complexities of maintaining system consistency, enforcing transactional integrity, and managing derived data views in environments prone to network failures and concurrency challenges.

The documentation is available in multiple formats, including an exportable digital book version, to support study and reference across various devices.

## Tags

### Education & Learning Resources

- [System Architecture Guides](https://awesome-repositories.com/f/education-learning-resources/educational-resources/reference-and-media/tutorials-media-curated-lists/technical-tutorials/systems-infrastructure/system-architecture-tutorials/system-architecture-guides.md) — Serves as a comprehensive technical reference for designing reliable, scalable, and maintainable data-intensive applications.

### Software Engineering & Architecture

- [Data System Design Principles](https://awesome-repositories.com/f/software-engineering-architecture/software-architecture/foundational-theory-and-guidance/software-architecture-principles/code-quality-maintenance-philosophies/maintainability-principles/data-system-design-principles.md) — Provides a comprehensive technical guide on the principles and trade-offs required to build modern data-intensive applications. ([source](https://cdn.jsdelivr.net/gh/Vonng/ddia@main/README.md))
- [Architectural Trade-offs](https://awesome-repositories.com/f/software-engineering-architecture/architectural-trade-offs.md) — Analyzes architectural implications of choosing between transactional and analytical systems, distributed versus single-node deployments, and cloud versus self-hosted infrastructure. ([source](https://ddia.vonng.com/part-i/))
- [Distributed Consensus Protocols](https://awesome-repositories.com/f/software-engineering-architecture/distributed-consensus-protocols.md) — Provides a comprehensive analysis of consensus protocols and coordination mechanisms for distributed systems.
- [Distributed Systems Architectures](https://awesome-repositories.com/f/software-engineering-architecture/distributed-systems-architectures.md) — Designs reliable and scalable systems by applying principles of data modeling, storage, and consistency.
- [Distributed Data Management](https://awesome-repositories.com/f/software-engineering-architecture/distributed-systems/distributed-data-management.md) — Covers fundamental strategies for data replication, partitioning, and consensus in distributed environments. ([source](https://ddia.vonng.com))
- [System Architecture Designs](https://awesome-repositories.com/f/software-engineering-architecture/software-architecture/architectural-patterns/backend-enterprise-systems/system-architecture-designs.md) — Evaluates fundamental trade-offs in reliability, scalability, and maintainability for data-intensive system architectures. ([source](https://ddia.vonng.com))
- [Software Design Principles](https://awesome-repositories.com/f/software-engineering-architecture/software-design-principles.md) — Guides fundamental trade-offs in data modeling, storage engines, and consistency models for modern infrastructure.
- [Commit Protocols](https://awesome-repositories.com/f/software-engineering-architecture/commit-protocols.md) — Coordinates multi-phase protocols to ensure global atomicity in distributed transactions. ([source](https://ddia.vonng.com/ch8/))
- [Distributed Fault Handling](https://awesome-repositories.com/f/software-engineering-architecture/distributed-system-challenges/distributed-fault-handling.md) — Addresses the complexities of maintaining system consistency and reliability in the face of network and node failures. ([source](https://ddia.vonng.com/part-ii/))
- [Event Sourcing](https://awesome-repositories.com/f/software-engineering-architecture/event-sourcing.md) — Implements architectural patterns that store state as an immutable sequence of events. ([source](https://ddia.vonng.com/ch3))
- [Fault Tolerance Implementation](https://awesome-repositories.com/f/software-engineering-architecture/fault-tolerance-strategies/fault-tolerance-implementation.md) — Details the implementation of fault tolerance to ensure system resilience despite component failures. ([source](https://ddia.vonng.com/ch2))
- [Data Processing Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/data-processing-pipelines.md) — Builds robust batch and stream processing workflows to transform and move data across multi-stage architectures.
- [Distributed Transaction Management](https://awesome-repositories.com/f/software-engineering-architecture/distributed-transaction-management.md) — Ensures data integrity across distributed nodes by coordinating multi-partition transactions. ([source](https://ddia.vonng.com/part-ii/))
- [Event Driven Messaging](https://awesome-repositories.com/f/software-engineering-architecture/event-driven-messaging.md) — Decouples services through message brokers and immutable event logs for reliable asynchronous communication.
- [Event Propagation Patterns](https://awesome-repositories.com/f/software-engineering-architecture/event-propagation-patterns.md) — Describes architectural patterns for propagating state changes through event logs to downstream systems. ([source](https://ddia.vonng.com/ch13))
- [Microservices Architectures](https://awesome-repositories.com/f/software-engineering-architecture/microservices-architectures.md) — Provides architectural patterns for decomposing complex applications into independent, scalable service units. ([source](https://ddia.vonng.com/ch1/))
- [Actor Models](https://awesome-repositories.com/f/software-engineering-architecture/actor-models.md) — Encapsulates logic into independent units that communicate via asynchronous messages for distributed state management. ([source](https://ddia.vonng.com/ch5))
- [Distributed Coordination Systems](https://awesome-repositories.com/f/software-engineering-architecture/distributed-coordination-systems.md) — Implements frameworks and protocols for managing consensus and state agreement across distributed nodes. ([source](https://ddia.vonng.com/glossary))
- [Idempotency Mechanisms](https://awesome-repositories.com/f/software-engineering-architecture/idempotency-mechanisms.md) — Uses unique request identifiers across distributed partitions to guarantee that operations execute exactly once. ([source](https://ddia.vonng.com/ch13))
- [Stateful Distributed Actors](https://awesome-repositories.com/f/software-engineering-architecture/stateful-distributed-actors.md) — Scales distributed actors across multiple nodes using location-independent messaging. ([source](https://ddia.vonng.com/ch5/))
- [State Change Broadcasting](https://awesome-repositories.com/f/software-engineering-architecture/architectural-design-patterns/state-management/reactive-subscription-systems/state-change-subscriptions/state-change-broadcasting.md) — Details methods for broadcasting state changes to maintain consistency across distributed nodes. ([source](https://ddia.vonng.com/ch10/))
- [Fault Tolerance Strategies](https://awesome-repositories.com/f/software-engineering-architecture/fault-tolerance-strategies.md) — Provides architectural guidance on eliminating single points of failure through redundancy and cluster-based design. ([source](https://ddia.vonng.com/ch2/))
- [Monotonic Time Measurement](https://awesome-repositories.com/f/software-engineering-architecture/monotonic-time-measurement.md) — Uses monotonic clocks to calculate durations and timeouts accurately without being affected by system clock adjustments. ([source](https://ddia.vonng.com/ch9))
- [Tail Latency Measurement](https://awesome-repositories.com/f/software-engineering-architecture/performance-reliability/performance-engineering/latency-optimization/tail-latency-measurement.md) — Explains how to measure tail latency to optimize performance for critical system requests. ([source](https://ddia.vonng.com/ch2/))
- [Retry Policies](https://awesome-repositories.com/f/software-engineering-architecture/retry-policies.md) — Implements retry logic with configurable exponential backoff strategies to handle transient failures. ([source](https://ddia.vonng.com/ch2/))
- [Workflow Orchestrators](https://awesome-repositories.com/f/software-engineering-architecture/workflow-orchestrators.md) — Covers the orchestration of persistent, multi-step workflows across distributed services. ([source](https://ddia.vonng.com/ch5))
- [Workflow Persistence](https://awesome-repositories.com/f/software-engineering-architecture/workflow-persistence.md) — Explores workflow persistence mechanisms for ensuring reliable, exactly-once task execution. ([source](https://ddia.vonng.com/ch5/))
- [Deterministic Simulation Environments](https://awesome-repositories.com/f/software-engineering-architecture/deterministic-simulation-environments.md) — Executes code in controlled environments to ensure reproducible results by eliminating non-deterministic inputs like time and latency. ([source](https://ddia.vonng.com/ch9/))
- [Fault Tolerance](https://awesome-repositories.com/f/software-engineering-architecture/fault-tolerance.md) — Simulates network failures and process crashes to verify system resilience under adverse conditions. ([source](https://ddia.vonng.com/ch9/))

### Data & Databases

- [Data Storage Architectures](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-persistence-storage/data-storage-architectures.md) — Provides a comprehensive guide to the principles and architectures of modern data systems. ([source](https://ddia.vonng.com/indexes/))
- [Unified Batch and Stream Processing Engines](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing-frameworks/unified-batch-and-stream-processing-engines.md) — Orchestrates data movement using unified engines for both batch and stream processing models. ([source](https://ddia.vonng.com))
- [Leaderless Replication](https://awesome-repositories.com/f/data-databases/data-synchronization/leaderless-replication.md) — Supports leaderless replication to eliminate single points of failure in the write path. ([source](https://ddia.vonng.com/ch6/))
- [Multi-Leader Replication](https://awesome-repositories.com/f/data-databases/fault-tolerant-storage/multi-leader-replication.md) — Provides multi-leader replication strategies to improve write availability and fault tolerance in distributed systems. ([source](https://ddia.vonng.com/ch6/))
- [Log Compaction](https://awesome-repositories.com/f/data-databases/log-processing-engines/log-compaction.md) — Reduce storage requirements by keeping only the latest value for each key in a log to enable efficient state reconstruction and snapshotting of historical data streams. ([source](https://ddia.vonng.com/ch12))
- [Single-Leader Replication](https://awesome-repositories.com/f/data-databases/primary-replica-replication/single-leader-replication.md) — Implements single-leader replication to maintain consistent state across distributed clusters. ([source](https://ddia.vonng.com/ch6/))
- [Change Data Capture](https://awesome-repositories.com/f/data-databases/change-data-capture.md) — Extracts database write events into streams to synchronize downstream systems. ([source](https://ddia.vonng.com/ch12/))
- [Database Design Patterns](https://awesome-repositories.com/f/data-databases/database-design-patterns.md) — Implements storage engines, indexing strategies, and transaction management for high-throughput data systems.
- [Distributed Transaction Processing](https://awesome-repositories.com/f/data-databases/distributed-transaction-processing.md) — Maintains atomicity and consistency across distributed systems using two-phase commit and consensus algorithms. ([source](https://ddia.vonng.com/indexes/))
- [Sorted Indexing](https://awesome-repositories.com/f/data-databases/hierarchical-index-sorting/sorted-indexing.md) — Organize data in sorted structures like tables or trees to enable efficient point lookups and range queries across large datasets. ([source](https://ddia.vonng.com/ch4/))
- [Log-Structured Storage](https://awesome-repositories.com/f/data-databases/log-structured-storage.md) — Organizes data into immutable, compressed chunks for efficient access in storage engines. ([source](https://ddia.vonng.com/ch4/))
- [Strong Consistency Guarantees](https://awesome-repositories.com/f/data-databases/strong-consistency-guarantees.md) — Ensures all clients see the most recent value for a data item immediately after a write. ([source](https://ddia.vonng.com/ch10))
- [Persistence & Durability](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-persistence-storage/persistence-durability.md) — Ensures data remains intact and accessible over time, even during system failures or interruptions. ([source](https://ddia.vonng.com/ch8))
- [Data Integration Tools](https://awesome-repositories.com/f/data-databases/data-integration-tools.md) — Automates the movement, transformation, and synchronization of data between disparate storage locations and distributed environments. ([source](https://ddia.vonng.com/ch13))
- [Unique Identifier Generators](https://awesome-repositories.com/f/data-databases/data-management/unique-identifier-generators.md) — Provides utilities for creating globally unique identifiers to ensure data consistency across distributed systems. ([source](https://ddia.vonng.com/ch10/))
- [Data Partitioning](https://awesome-repositories.com/f/data-databases/data-partitioning.md) — Distributes large datasets into smaller segments across nodes to enable horizontal scaling. ([source](https://ddia.vonng.com/ch7/))
- [Data Processing Pipelines](https://awesome-repositories.com/f/data-databases/data-processing-pipelines.md) — Provides systems and workflows for ingesting, transforming, and orchestrating high-throughput data processing tasks. ([source](https://ddia.vonng.com/v1_tw/))
- [Data Stream Processors](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing-frameworks/data-stream-processors.md) — Executes complex transformations on real-time data flows through batch or streaming processing tasks. ([source](https://ddia.vonng.com/ch12))
- [Exactly-Once Processing Semantics](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing-frameworks/exactly-once-processing-semantics.md) — Ensures operations produce the same final state despite retries by using idempotent logic and unique request identifiers. ([source](https://ddia.vonng.com/ch13/))
- [Distributed Computing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/distributed-processing-frameworks/distributed-computing.md) — Provides frameworks for executing large-scale data processing and computation across distributed clusters. ([source](https://ddia.vonng.com/ch1))
- [Data Transformation](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation.md) — Provides tools and utilities for modifying, restructuring, or converting raw data into desired formats and schemas. ([source](https://ddia.vonng.com/part-iii/))
- [Data Encoding and Serialization](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/data-encoding-serialization.md) — Provides libraries for converting data between binary, text, and portable interchange formats for storage or transmission. ([source](https://ddia.vonng.com/ch5/))
- [Stream Processing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/stream-processing-systems/stream-processing.md) — Executes continuous stream processing workflows to derive real-time insights. ([source](https://ddia.vonng.com/indexes/))
- [Data Replication](https://awesome-repositories.com/f/data-databases/data-replication.md) — Synchronizes data across distributed nodes to ensure availability and fault tolerance. ([source](https://ddia.vonng.com/glossary/))
- [Data Schema Management](https://awesome-repositories.com/f/data-databases/data-schema-management.md) — Uses interface definition languages to specify data structures for consistent encoding across systems. ([source](https://ddia.vonng.com/ch5/))
- [Data Sharding](https://awesome-repositories.com/f/data-databases/data-sharding.md) — Covers partitioning and sharding strategies for scaling data systems horizontally.
- [Data Snapshotting](https://awesome-repositories.com/f/data-databases/data-snapshotting.md) — Provides mechanisms for creating point-in-time backups of database state. ([source](https://ddia.vonng.com/ch8/))
- [Authoritative](https://awesome-repositories.com/f/data-databases/data-sources/authoritative.md) — Designates primary systems as the authoritative source of truth for all other system components. ([source](https://ddia.vonng.com/part-iii/))
- [Data Warehousing](https://awesome-repositories.com/f/data-databases/data-warehousing.md) — Provides platforms designed for large-scale data storage and high-performance analytical query execution. ([source](https://ddia.vonng.com/ch1/))
- [Atomic Transactions](https://awesome-repositories.com/f/data-databases/database-management-systems/database-systems-management/connection-transaction-management/atomic-transactions.md) — Executes multiple data manipulation operations as a single unit to ensure consistency. ([source](https://ddia.vonng.com/ch8/))
- [Distributed Data Management](https://awesome-repositories.com/f/data-databases/distributed-data-management.md) — Scales data across nodes using replication and partitioning for workload management. ([source](https://ddia.vonng.com/part-ii/))
- [Full Text Search](https://awesome-repositories.com/f/data-databases/full-text-search.md) — Builds inverted indexes to enable efficient keyword-based retrieval and complex text matching within large document collections. ([source](https://ddia.vonng.com/ch4))
- [Horizontal Scaling Architectures](https://awesome-repositories.com/f/data-databases/horizontal-scaling-architectures.md) — Describes horizontal scaling architectures that partition data and distribute load across multiple nodes. ([source](https://ddia.vonng.com/ch2/))
- [Materialized Views](https://awesome-repositories.com/f/data-databases/materialized-views.md) — Maintains pre-computed views to provide near-instant responses for frequently accessed aggregate metrics. ([source](https://ddia.vonng.com/ch4/))
- [Multi-Tenant Data Management](https://awesome-repositories.com/f/data-databases/multi-tenant-data-management.md) — Provides systems for isolating data by tenant, including automated ID assignment and administrative interfaces. ([source](https://ddia.vonng.com/ch7/))
- [Probabilistic Data Structures](https://awesome-repositories.com/f/data-databases/probabilistic-data-structures.md) — Use probabilistic data structures to quickly determine if a key is absent from a storage segment to reduce unnecessary disk input-output during read operations. ([source](https://ddia.vonng.com/ch4))
- [Relational Data Modeling](https://awesome-repositories.com/f/data-databases/relational-data-modeling.md) — Provides foundational techniques for organizing data into structured tables with defined relationships. ([source](https://ddia.vonng.com/ch3/))
- [Secondary Indexes](https://awesome-repositories.com/f/data-databases/secondary-indexes.md) — Creates auxiliary lookup structures on non-primary columns to enable efficient searching and filtering across datasets. ([source](https://ddia.vonng.com/ch4))
- [Derived Data Generation](https://awesome-repositories.com/f/data-databases/secondary-indexes/derived-data-generation.md) — Creates secondary datasets from primary records through repeatable processes to optimize read performance. ([source](https://ddia.vonng.com/glossary/))
- [Log-Structured Merge-Trees](https://awesome-repositories.com/f/data-databases/storage-engines/b-tree/log-structured-merge-trees.md) — Uses log-structured merge-trees to handle high-throughput write workloads efficiently. ([source](https://ddia.vonng.com/ch4))
- [Transaction Management](https://awesome-repositories.com/f/data-databases/transaction-management.md) — Handles database transaction lifecycles, including isolation levels, commit protocols, and rollback operations. ([source](https://ddia.vonng.com/ch8/))
- [Analytical Query Engines](https://awesome-repositories.com/f/data-databases/analytical-query-engines.md) — Provides systems designed for high-speed aggregation and complex reporting on large datasets. ([source](https://ddia.vonng.com/ch4/))
- [Write Skew Mitigations](https://awesome-repositories.com/f/data-databases/concurrent-write-optimizations/write-skew-mitigations.md) — Prevents data corruption by mitigating write skew in concurrent transaction environments. ([source](https://ddia.vonng.com/ch8/))
- [Data Authority Models](https://awesome-repositories.com/f/data-databases/data-access-querying/data-authority-models.md) — Distinguishes between authoritative systems of record and derived data views. ([source](https://ddia.vonng.com/part-iii))
- [Data Analytics Engines](https://awesome-repositories.com/f/data-databases/data-analysis-visualization/analytical-platforms-engines/data-analytics-engines.md) — Executes complex aggregations and analytical queries on large volumes of data. ([source](https://ddia.vonng.com/glossary/))
- [Data Constraints](https://awesome-repositories.com/f/data-databases/data-constraints.md) — Maintains system-wide integrity by enforcing unique key registration and data constraints atomically across nodes. ([source](https://ddia.vonng.com/ch10))
- [Uniqueness Enforcement](https://awesome-repositories.com/f/data-databases/data-management/unique-identifier-generators/uniqueness-enforcement.md) — Prevents conflicting operations by requiring a single, agreed-upon state across all nodes for unique constraints. ([source](https://ddia.vonng.com/ch10/))
- [Automatic Rebalancing](https://awesome-repositories.com/f/data-databases/data-partitioning-strategies/automatic-rebalancing.md) — Adjusts data distribution automatically in response to load changes or cluster membership updates. ([source](https://ddia.vonng.com/ch7/))
- [Data Storage Optimizers](https://awesome-repositories.com/f/data-databases/data-storage-optimizers.md) — Implements indexing strategies and storage engines tailored for transactional and analytical workloads. ([source](https://ddia.vonng.com//part-i))
- [Database Performance Optimizers](https://awesome-repositories.com/f/data-databases/database-performance-optimizers.md) — Minimizes disk input-output operations during data retrieval tasks using caching and indexing techniques. ([source](https://ddia.vonng.com/ch4/))
- [Distributed Filesystems](https://awesome-repositories.com/f/data-databases/distributed-filesystems.md) — Distributes large files across nodes to enable high-throughput access and fault tolerance. ([source](https://ddia.vonng.com/ch11))
- [Document Stores](https://awesome-repositories.com/f/data-databases/document-stores.md) — Stores information as flexible JSON documents to accommodate semi-structured data. ([source](https://ddia.vonng.com/ch3/))
- [Fault Tolerant Storage](https://awesome-repositories.com/f/data-databases/fault-tolerant-storage.md) — Distributes large datasets across nodes using fault-tolerant storage systems. ([source](https://ddia.vonng.com/ch11/))
- [Graph Data Models](https://awesome-repositories.com/f/data-databases/graph-data-models.md) — Represents entities as vertices and relationships as edges to query interconnected data structures. ([source](https://ddia.vonng.com/ch3/))
- [Resource Scaling Strategies](https://awesome-repositories.com/f/data-databases/horizontal-database-scaling/resource-scaling-strategies.md) — Explains strategies for scaling system resources to maintain performance as load grows. ([source](https://ddia.vonng.com/ch2))
- [Hybrid Transactional Analytical Platforms](https://awesome-repositories.com/f/data-databases/hybrid-transactional-analytical-platforms.md) — Maintains distinct stores for transactional and analytical workloads to optimize performance. ([source](https://ddia.vonng.com/ch1/))
- [Indexing and Search](https://awesome-repositories.com/f/data-databases/indexing-and-search.md) — Implements indexing strategies to accelerate data access across diverse query types. ([source](https://ddia.vonng.com/part-i/))
- [Message Brokers](https://awesome-repositories.com/f/data-databases/message-brokers.md) — Implements message brokers that utilize durable append-only logs to support asynchronous communication and historical data replay. ([source](https://ddia.vonng.com/ch12))
- [Stream Analytics Processing](https://awesome-repositories.com/f/data-databases/real-time-analytics/stream-analytics-processing.md) — Details the mechanics of performing stream analytics to derive insights from evolving datasets. ([source](https://ddia.vonng.com/ch12/))
- [Complex Data Modeling](https://awesome-repositories.com/f/data-databases/relational-data-modeling/complex-data-modeling.md) — Organizes information using diverse structures to support complex application query requirements. ([source](https://ddia.vonng.com/toc/))
- [Data Indexing Strategies](https://awesome-repositories.com/f/data-databases/search-indexing-technologies/search-indexing/data-indexing-strategies.md) — Organizes data structures for efficient retrieval and search performance. ([source](https://ddia.vonng.com/ch4))
- [Analytical Data Engines](https://awesome-repositories.com/f/data-databases/analytical-data-engines.md) — Provides high-performance engines for processing and analyzing large-scale tabular datasets. ([source](https://ddia.vonng.com/ch3))
- [Batch Data Processing](https://awesome-repositories.com/f/data-databases/batch-data-processing.md) — Processes large, fixed datasets as single units without modifying original input data. ([source](https://ddia.vonng.com/glossary))
- [Data Extraction](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-extraction-ingestion/data-extraction.md) — Automates the extraction and transformation of data from source systems into analytical warehouses. ([source](https://ddia.vonng.com/glossary))
- [Schema Evolution](https://awesome-repositories.com/f/data-databases/data-governance-modeling/data-modeling-schemas/schema-evolution.md) — Manages schema changes and format transitions to maintain system compatibility over time. ([source](https://ddia.vonng.com/part-i/))
- [Data Joins](https://awesome-repositories.com/f/data-databases/data-joins.md) — Provides mechanisms for merging data streams and tables based on shared keys to enable complex correlations. ([source](https://ddia.vonng.com/ch12/))
- [ETL Workflows](https://awesome-repositories.com/f/data-databases/data-pipeline-orchestration/etl-workflows.md) — Maintains data pipelines by performing parallel extraction, transformation, and loading of data. ([source](https://ddia.vonng.com/ch11))
- [Dataframe Processing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/dataframe-processing.md) — Manipulates tabular datasets through programmatic transformations for statistical analysis. ([source](https://ddia.vonng.com/ch3/))
- [Structured Event Streams](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/stream-processing-systems/data-streaming/structured-event-streams.md) — Details the streaming of event data to enable continuous, low-latency processing. ([source](https://ddia.vonng.com/ch12))
- [Data Shuffling Algorithms](https://awesome-repositories.com/f/data-databases/data-shuffling-algorithms.md) — Provides methods for redistributing partitioned data across nodes to ensure related records are grouped for processing. ([source](https://ddia.vonng.com/ch11/))
- [Data Synchronization](https://awesome-repositories.com/f/data-databases/data-synchronization.md) — Synchronizes redundant data representations to maintain consistency with the authoritative source of truth. ([source](https://ddia.vonng.com/ch1/))
- [Dataframe Engines](https://awesome-repositories.com/f/data-databases/dataframe-engines.md) — Provides engines for cleaning and transforming tabular data using dataframe abstractions. ([source](https://ddia.vonng.com/ch3))
- [MapReduce Processing Engines](https://awesome-repositories.com/f/data-databases/mapreduce-processing-engines.md) — Executes batch computations by mapping input records to key-value pairs and reducing them into aggregated results. ([source](https://ddia.vonng.com/ch11))
- [Object Storage](https://awesome-repositories.com/f/data-databases/object-storage.md) — Stores data as immutable objects to provide a scalable alternative to traditional file systems. ([source](https://ddia.vonng.com/ch11))
- [Query Languages](https://awesome-repositories.com/f/data-databases/query-languages.md) — Supports specialized query languages for traversing hierarchical and recursive data relationships. ([source](https://ddia.vonng.com/ch3/))
- [Search and Indexing](https://awesome-repositories.com/f/data-databases/search-indexing-technologies/search-indexing/search-and-indexing.md) — Provides specialized indexing structures to enable complex spatial, full-text, and semantic queries. ([source](https://ddia.vonng.com/ch4/))
- [Global Indexes](https://awesome-repositories.com/f/data-databases/secondary-indexes/global-indexes.md) — Maintains partitioned indexes across all shards to enable efficient lookups for specific values. ([source](https://ddia.vonng.com/ch7/))
- [Local Indexes](https://awesome-repositories.com/f/data-databases/secondary-indexes/local-indexes.md) — Indexes data within each partition independently to allow efficient local lookups without cross-node coordination. ([source](https://ddia.vonng.com/ch7/))
- [B-Tree](https://awesome-repositories.com/f/data-databases/storage-engines/b-tree.md) — Implements B-Tree storage structures to optimize disk-based range queries and efficient data lookups. ([source](https://ddia.vonng.com/ch4))
- [Stored Procedures](https://awesome-repositories.com/f/data-databases/stored-procedures.md) — Executes transaction logic directly within the database engine to improve latency and throughput. ([source](https://ddia.vonng.com/glossary/))
- [Caching](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/caching-performance/caching.md) — Stores frequently accessed data in temporary memory to reduce latency and improve application performance. ([source](https://ddia.vonng.com/glossary))
- [Real-Time Data Streaming](https://awesome-repositories.com/f/data-databases/real-time-data-streaming.md) — Matches incoming documents against stored queries in real time to notify users when data meets specific criteria. ([source](https://ddia.vonng.com/ch12/))

### Networking & Communication

- [Consensus and Coordination Systems](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-computing/consensus-coordination-systems.md) — Coordinates agreement among distributed nodes to maintain consistent state across clusters. ([source](https://ddia.vonng.com/part-ii))
- [Data Synchronization and Consistency](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-computing/data-synchronization-consistency.md) — Coordinates state across distributed nodes to ensure consistency despite network failures and clock drift. ([source](https://ddia.vonng.com/part-ii/))
- [Distributed Systems Coordination](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-systems-coordination.md) — Coordinates state and consensus among nodes to resolve distributed system faults. ([source](https://ddia.vonng.com/toc/))
- [Node Failure Detection](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-systems-coordination/distributed-systems-configuration/heartbeat-and-timeout-configurations/node-failure-detection.md) — Explains how to detect node failures and trigger automated recovery in distributed systems. ([source](https://ddia.vonng.com/ch10/))
- [Message Brokers](https://awesome-repositories.com/f/networking-communication/message-brokers.md) — Facilitates asynchronous communication and decoupling between distributed services using message brokers. ([source](https://ddia.vonng.com/ch5/))
- [Server-to-Client Event Emissions](https://awesome-repositories.com/f/networking-communication/communication-protocols-architectures/communication-paradigms/server-to-client-event-emissions.md) — Streams server-side state changes to clients in real-time to reduce polling. ([source](https://ddia.vonng.com/ch13/))
- [Distributed Coordination Services](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-systems-coordination/distributed-systems-infrastructure/distributed-coordination-services.md) — Provides infrastructure services for coordinating distributed nodes through leader election and atomic operations. ([source](https://ddia.vonng.com/ch10))
- [Message Passing](https://awesome-repositories.com/f/networking-communication/message-passing.md) — Connects producers and consumers via message brokers to support asynchronous communication and publish-subscribe patterns. ([source](https://ddia.vonng.com/ch12/))
- [Request Routing](https://awesome-repositories.com/f/networking-communication/network-infrastructure-routing/network-routing-traffic-management/request-routing.md) — Directs client queries to specific nodes using routing layers and partition-aware clients. ([source](https://ddia.vonng.com/ch7/))
- [Publish-Subscribe Systems](https://awesome-repositories.com/f/networking-communication/publish-subscribe-systems.md) — Details the implementation of publish-subscribe messaging systems for reliable event distribution. ([source](https://ddia.vonng.com/ch12))
- [Message Ordering Guarantees](https://awesome-repositories.com/f/networking-communication/network-reliability-diagnostics/reliability-scaling/message-ordering-guarantees.md) — Discusses the importance of message ordering guarantees for maintaining consistency in distributed event-driven systems. ([source](https://ddia.vonng.com/ch10))

### Artificial Intelligence & ML

- [Event-Sourced Provenance](https://awesome-repositories.com/f/artificial-intelligence-ml/memory-provenance-tracking/event-sourced-provenance.md) — Explains how to audit data provenance using immutable event logs for reliable recovery. ([source](https://ddia.vonng.com/ch13/))
- [Vector Similarity Search](https://awesome-repositories.com/f/artificial-intelligence-ml/vector-similarity-search.md) — Performs similarity searches on high-dimensional vector embeddings using specialized indexing structures. ([source](https://ddia.vonng.com/ch4))

### DevOps & Infrastructure

- [Distributed Locks](https://awesome-repositories.com/f/devops-infrastructure/distributed-locks.md) — Uses atomic operations to ensure only one node can hold a lease or lock at a time. ([source](https://ddia.vonng.com/ch10))
- [Distributed Deployment Patterns](https://awesome-repositories.com/f/devops-infrastructure/distributed-systems/distributed-deployment-patterns.md) — Details strategies for deploying applications across multiple nodes to achieve high availability and scalability. ([source](https://ddia.vonng.com/ch1/))
- [Partitioning Algorithms](https://awesome-repositories.com/f/devops-infrastructure/load-balancing/partitioning-algorithms.md) — Uses key-range or hash-based partitioning to distribute data load evenly across nodes. ([source](https://ddia.vonng.com/ch7))
- [API Throttling](https://awesome-repositories.com/f/devops-infrastructure/api-throttling.md) — Rejects or delays requests during peak load to prevent cascading failures and maintain stability. ([source](https://ddia.vonng.com/ch2))
- [Distributed Task Orchestration](https://awesome-repositories.com/f/devops-infrastructure/distributed-task-orchestration.md) — Schedules and executes tasks across clusters by managing resource allocation and monitoring. ([source](https://ddia.vonng.com/ch11))
- [Event Pattern Detection](https://awesome-repositories.com/f/devops-infrastructure/event-triggers/event-pattern-detection.md) — Describes techniques for detecting event patterns in streams to trigger automated actions. ([source](https://ddia.vonng.com/ch12/))
- [Rate Limiters](https://awesome-repositories.com/f/devops-infrastructure/rate-limiters.md) — Limits incoming traffic rates to prevent system saturation and cascading failures. ([source](https://ddia.vonng.com/ch2/))
- [Kernel Fault Injection](https://awesome-repositories.com/f/devops-infrastructure/fault-tolerance/kernel-fault-injection.md) — Covers the practice of injecting faults to verify system resilience against real-world outages. ([source](https://ddia.vonng.com/ch2/))
- [Fault Injection Testing](https://awesome-repositories.com/f/devops-infrastructure/fault-tolerance/kernel-fault-injection/fault-injection-testing.md) — Discusses the use of fault injection to verify the effectiveness of fault-tolerance mechanisms. ([source](https://ddia.vonng.com/ch2))
- [Local-First Synchronization](https://awesome-repositories.com/f/devops-infrastructure/local-first-synchronization.md) — Discusses replacing synchronous requests with event streams to maintain local state and improve robustness. ([source](https://ddia.vonng.com/ch13))
- [Performance Metrics](https://awesome-repositories.com/f/devops-infrastructure/system-requirements/performance-metrics.md) — Establishes metrics for reliability, scalability, and maintainability to ensure systems meet performance expectations. ([source](https://ddia.vonng.com//part-i))

### Security & Cryptography

- [Concurrency Controls](https://awesome-repositories.com/f/security-cryptography/injection-prevention/concurrency-controls.md) — Prevents lost updates in concurrent transactions using atomic operations and conflict detection. ([source](https://ddia.vonng.com/ch8/))
- [Business Constraint Enforcement](https://awesome-repositories.com/f/security-cryptography/identity-access-management/access-control/business-constraint-enforcement.md) — Validates business rules by routing conflicting operations to partitioned logs for sequential, deterministic processing. ([source](https://ddia.vonng.com/ch13/))

### Business & Productivity Software

- [Analytical Schemas](https://awesome-repositories.com/f/business-productivity-software/business-intelligence-strategy/business-intelligence-analytics/analytical-schemas.md) — Implements star and snowflake schema patterns to accelerate reporting and analytical queries. ([source](https://ddia.vonng.com/ch3/))

### Programming Languages & Runtimes

- [Directed Acyclic Graph Execution Engines](https://awesome-repositories.com/f/programming-languages-runtimes/runtime-execution-environments/runtime-environments/runtimes/graph-symbolic-execution-engines/directed-acyclic-graph-execution-engines.md) — Models data movement as directed acyclic graphs to optimize execution plans and ensure repeatable transformations.

### Scientific & Mathematical Computing

- [Architectural Abstractions](https://awesome-repositories.com/f/scientific-mathematical-computing/numerical-mathematical-foundations/algorithms-and-complexity/architectural-abstractions.md) — Uses abstractions to hide implementation details and reduce cognitive load in complex systems. ([source](https://ddia.vonng.com/ch2))

### System Administration & Monitoring

- [System Requirement Definitions](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/system-requirement-definitions.md) — Establishes metrics for reliability, scalability, and maintainability to guide system design. ([source](https://ddia.vonng.com/part-i/))
- [Operational Task Automation](https://awesome-repositories.com/f/system-administration-monitoring/operational-task-automation.md) — Automates system maintenance and recovery tasks to reduce manual intervention in large-scale environments. ([source](https://ddia.vonng.com/ch2/))

### Development Tools & Productivity

- [Pipeline Execution Engines](https://awesome-repositories.com/f/development-tools-productivity/pipeline-execution-engines.md) — Executes multi-stage analytical workflows by modeling data movement between operators and optimizing execution plans. ([source](https://ddia.vonng.com/ch11))
- [Flow Control](https://awesome-repositories.com/f/development-tools-productivity/plugin-management/backpressure-management/flow-control.md) — Forces data senders to reduce transmission rates when receiving systems are overloaded. ([source](https://ddia.vonng.com/glossary/))

### Testing & Quality Assurance

- [System Execution Verifiers](https://awesome-repositories.com/f/testing-quality-assurance/debugging-diagnostics/runtime-debugging-tools/runtime-verification/system-execution-verifiers.md) — Validates distributed algorithms against formal models to ensure correctness across all execution states. ([source](https://ddia.vonng.com/ch9/))

### Web Development

- [Data View Projections](https://awesome-repositories.com/f/web-development/backend-to-frontend-bridges/data-view-projections.md) — Explains how to project and expose specific data views to meet client-side requirements. ([source](https://ddia.vonng.com/ch3/))