citusdata/pg_shard
Pg Shard
该项目是一个数据库扩展，支持跨多个 PostgreSQL 节点的水平扩展和分布式数据分区。它作为一个中间件解决方案，将大型数据库表分布在集群中，以增加总存储容量并提高不断增长的数据集的写入性能。
该系统采用基于哈希的分区将表记录分布在物理节点上，并使用中央元数据目录来跟踪分片位置。它提供透明的查询路由，将标准 SQL 命令从协调节点引导至适当的工作节点，而无需修改现有的应用逻辑。
该框架包含通过分片复制和自动一致性修复过程来维护数据冗余和集群完整性的工具。它还支持高吞吐量数据摄入，利用并发处理将海量数据集导入分布式表中。
Features
- PostgreSQL Sharding Extensions - Enables horizontal scaling and distributed data partitioning across multiple PostgreSQL nodes for improved write performance.
- Coordinator-Worker Topologies - Directs incoming SQL commands from a central entry point to the appropriate backend nodes based on metadata.
- Database Middleware - Routes SQL queries to appropriate worker nodes to provide transparent horizontal scaling for existing database applications.
- Database Sharding - Partitions data across a cluster of servers to manage massive workloads that exceed the capacity of a single machine.
- Query Routing - Automatically routes standard database commands from a central node to the appropriate worker nodes without requiring modifications to existing application logic.
- Horizontal Database Scaling - Distributes large database tables across multiple nodes to increase storage capacity and improve write performance for growing datasets.
- Horizontal Table Partitionings - Distributes table records across multiple nodes using hash partitioning to achieve horizontal scaling and expand total storage capacity.
- Hash-Based Data Distribution - Distributes table rows across multiple physical nodes by applying a mathematical function to a specific column value.
- Bulk Data Ingestion - Supports high-throughput ingestion of massive datasets into distributed tables using concurrent processes to maximize write performance.
- Cross-Shard Data Replication - Maintains multiple copies of table shards across different worker nodes to ensure data redundancy and high availability during node outages.
- Database Sharding Solutions - Provides a solution for managing shard replicas and data redundancy to ensure high availability and consistency across a distributed database cluster.
- Distributed Sharding Architectures - Maintains multiple copies of data segments across different physical servers to ensure high availability and fault tolerance.
- External File Imports - Enables importing large datasets from external files into distributed tables using parallel processing to achieve high write throughput.
- Horizontal Scaling - Distributes large database tables across a cluster of nodes to increase total storage capacity and throughput.
- Parallelized Ingestion - Loads massive volumes of information into a database cluster using concurrent processes to maximize throughput and reduce import times.
- Asynchronous Replica Synchronizations - Synchronizes data from healthy shard replicas to inconsistent locations to maintain cluster integrity after system errors or hardware failures.
- High Availability Clusters - Maintains redundant copies of data across multiple nodes to ensure continuous service and prevent data loss during hardware failures.
- Metadata-Driven Cluster Managements - Tracks the physical location of data shards using a central catalog to coordinate operations across the distributed system.
- Consistency Repairs - Synchronizes data from healthy shard replicas to inconsistent locations to maintain cluster integrity and ensure data accuracy.
