ydb-platform/ydb
Ydb
YDB is a distributed SQL database and analytical engine designed for horizontal scalability and strong consistency. It functions as a multi-model system that supports transactional and analytical workloads through a distributed architecture providing serializable ACID transactions.
The system is distinguished by its broad protocol compatibility, implementing the PostgreSQL wire protocol for standard SQL drivers and the Kafka protocol for messaging and streaming. It further serves as a vector database, supporting vector indexes and approximate nearest neighbor searches for semantic search and embeddings.
The platform manages data using a hybrid storage model with both row-oriented and column-oriented formats, utilizing vectorized query execution for petabyte-scale analytics. Its operational surface includes change data capture streaming, exactly-once persistent queues, and multi-zone high availability.
Deployment and lifecycle management are supported through a Kubernetes operator and infrastructure-as-code provisioning.
Features
- Analytical Data Engines - Processes petabyte-scale analytical queries using a vectorized query engine and columnar storage.
- Distributed SQL Databases - Provides a distributed SQL database architecture that combines horizontal scalability with strong transactional consistency.
- ACID Transaction Semantics - Guarantees serializable ACID transactions for strict data consistency across distributed nodes.
- Analytical Query Engines - Scales analytical workloads to petabytes using a vectorized query engine and distributed execution planner.
- Hybrid Storage Formats - Maintains data in both row and columnar formats to optimize for both transactional and analytical workloads.
- Cross-Zone Data Distribution - Distributes data across multiple availability zones to ensure access during simultaneous zone and rack failures.
- Data Recovery - Automatically restores data redundancy and service availability following disk, node, or datacenter failures.
- Declarative Query Languages - Provides a structured query language for declarative data retrieval and manipulation across a distributed cluster.
- Distributed Sharding Architectures - Implements distributed tablet sharding to enable horizontal scaling and balanced load distribution across nodes.
- Horizontal Scaling - Enables horizontal resource scaling by adding storage and computation nodes to the cluster.
- PostgreSQL-Compatible Databases - Implements the PostgreSQL wire protocol and SQL dialect for compatibility with standard database drivers.
- Real-Time Data Streaming - Integrates database tables with Kafka topics and processes unbounded data streams via change data capture.
- SQL Query Dialects - Provides a structured query language dialect for precise data management and retrieval.
- Vector Databases - Serves as a vector database supporting vector indexes and approximate nearest neighbor searches for embeddings.
- Vectorized Execution Engines - Utilizes a vectorized execution engine to process data in batches for high-performance analytical workloads.
- Column-Oriented Disk Storage - Organizes data independently by column to optimize analytical aggregates and maximize compression ratios.
- Distributed Data Workload Scaling - Distributes data and workloads across multiple nodes to ensure horizontal scalability and high availability.
- Hash-Based Data Distribution - Uses hash-based partitioning of keys to distribute data across shards and prevent insertion hotspots.
- Shard Rebalancing - Supports online addition or removal of nodes with automatic sharding to adapt to changing workloads.
- Backup & Recovery - Creates point-in-time database copies on object stores or filesystems for disaster recovery.
- Change Data Capture Streams - Provides real-time change data capture streaming of row-level modifications for external downstream processing.
- Erasure Coding - Uses erasure coding to provide high fault tolerance and data availability with lower storage overhead than mirroring.
- Tabular Row Storage - Stores data sequentially by row to optimize for transactional queries and fast single-record access.
- Database Wire Protocol Emulations - Implements various database wire protocols to allow standard drivers to interact with the internal system.
- Distributed Query Stream Processors - Runs streaming queries that automatically restart on failure and use checkpoints to persist state.
- Federated Data Query Engines - Executes federated queries to retrieve and process data from multiple external sources without migration.
- Indexing and Search - Supports a combination of primary, secondary, and vector indexes to optimize diverse data retrieval patterns.
- Multi-Tenant Resource Isolation - Runs multiple isolated databases within a single cluster using a shared storage layer to partition resources.
- PostgreSQL Protocol Compatibility - Supports the standard PostgreSQL wire protocol to ensure compatibility with existing PostgreSQL database drivers.
- Transactional Delivery Guarantees - Moves data between messaging topics and tables using transactional guarantees to ensure reliability.
- Vector Similarity Search - Executes approximate k-nearest neighbor queries on embeddings for semantic search and recommendations.
- Database Deployment Automation - Automates the installation and configuration of distributed database clusters across servers using orchestration tools.
- Operator-Managed Deployments - Employs a Kubernetes operator to automate the installation and lifecycle of storage and dynamic nodes.
- Database Cluster Deployments - Provides specialized orchestration for deploying and scaling distributed database clusters within container environments.
- Database Deployments - Offers automated deployment and management of database clusters in Kubernetes using a dedicated operator.
- Kafka Protocol Implementations - Implements the Kafka wire protocol to enable seamless integration with standard Kafka clients and topics.
- Database Protocol Implementations - Integrates with applications using the Kafka protocol for seamless messaging and streaming within the database.
- Persistent Queues - Provides persistent messaging queues with exactly-once delivery and automatic partitioning.
- Dynamic Shard Splitting - Automatically splits table partitions based on data size or request load to balance traffic across nodes.
