Shardingsphere

ShardingSphere is a distributed SQL database middleware that provides sharding, read-write splitting, and distributed transaction management for relational databases. It functions as a layer that intercepts SQL queries to distribute data across multiple physical database instances for horizontal scaling.

The project is distinguished by its ability to operate as either a standalone transparent database proxy or via direct integration as a JDBC driver. It features a SQL dialect translator that parses queries into abstract syntax trees to convert syntax between different database engines, enabling a unified interface across heterogeneous storage backends. Additionally, it includes a security gateway for transparent data encryption, dynamic data masking, and SQL firewall access control.

The system covers a broad range of data distribution and governance capabilities, including horizontal and vertical sharding, read-write splitting, and real-time data migration using change data capture pipelines. It also provides tools for database stress testing through shadow database routing and observability features for SQL execution tracing and performance monitoring.

The project is implemented in Java and supports deployment as a standalone cluster or as an embedded library.

Features

Database Partitioning and Sharding - Distributes datasets across multiple database instances using configurable horizontal and vertical sharding strategies.
Database Sharding - Implements architectural patterns for partitioning large datasets across multiple database nodes for horizontal scaling.
Distributed SQL Databases - Provides a unified interface for executing and translating SQL queries across heterogeneous distributed database clusters.
Distributed SQL Querying - Manages the distributed execution of SQL queries, sharding rules, and scaling configurations across a cluster.
Traffic Routing - Directs read and write operations between master and replica nodes to balance load and ensure high availability.
Sharding Middleware - Provides a middleware layer for sharding, read-write splitting, and distributed transaction management.
SQL Abstract Syntax Tree Parsing - Analyzes SQL statements into structured abstract syntax trees to implement custom routing and execution logic.
SQL Dialect Adapters - Translates SQL syntax between different database engines to provide a unified interface across heterogeneous storage backends.
AST-Based Rewriting - Implements AST-based SQL rewriting to transform queries for distributed routing and execution across physical database instances.
Database Proxying - Provides a standalone proxy that routes and optimizes database queries between clients and storage nodes.
Distributed Configuration Management - Synchronizes database settings and sharding rules dynamically across a cluster using a centralized configuration system.
High Availability Clusters - Ensures continuous service availability through cluster deployment, health checks, and automated failover.
Transparent Proxies - Operates as a transparent database proxy that intercepts standard protocols to execute distributed logic without application changes.
Data Protection - Provides a middleware layer for protecting sensitive information through transparent encryption and dynamic data masking.
Centralized Configuration Systems - Provides a centralized configuration system to distribute sharding rules and cluster metadata across distributed nodes.
Distributed Transaction Coordinators - Coordinates global transaction lifecycles across multiple database nodes to ensure atomicity and data consistency.
Distributed Transaction Management - Maintains global data consistency by coordinating transactions across multiple database nodes.
Cluster Management - Manages cluster metadata, instance registration, and distributed locking through a centralized registry.
Change Data Capture - Captures and applies incremental data changes in real-time for database migration and consistency.
Database Synchronization - Synchronizes data between different database systems to maintain consistency across disparate environments.
Data Migration - Moves data between different database instances using change data capture and automated migration pipelines.
Data Migration and Synchronization - Captures and applies incremental data changes in real-time to maintain consistency during migrations.
Database Drivers - Integrates sharding and encryption logic directly into applications as a JDBC driver.
CDC Synchronization - Moves data between different database systems using change data capture to maintain consistency.
Embedded Database Middleware - Embeds sharding and routing logic directly into applications as a lightweight library.
Read Replicas - Directs write operations to master nodes and distributes read requests across slave replicas to optimize performance.
Shadow Database Routing - Diverts specific traffic patterns to isolated shadow databases for real-world stress testing.
Shadow Database Isolation - Creates a shadow database environment to perform stress testing with real data without impacting production traffic.
Transparent Encryption Wrappers - Encrypts sensitive information at the middleware layer before it is stored in the underlying physical database.
Data Masking Tools - Hides sensitive data from users based on customizable masking algorithms without altering the stored data.
Transparent Data - Provides a middleware encryption layer that transparently encrypts or masks sensitive data fields before storage.
Database - Provides a security layer for transparent data encryption, dynamic data masking, and SQL firewalling.
SQL Firewalls - Implements an SQL firewall to enforce fine-grained permissions and restrict unauthorized database actions.
Pluggable Service Provider Interfaces - Allows customization of sharding algorithms and execution behaviors through a pluggable service provider interface.
Modular Provider Interfaces - Uses a modular provider interface to allow custom implementations of sharding algorithms and communication protocols.
Database Performance Monitors - Collects metrics and tracing data through observability hooks to analyze system health and query latency.
Distributed Tracing and Execution Analysis - Tracks SQL parsing and execution traces to identify bottlenecks and analyze request flows across distributed nodes.
Database Shadow Testing - Routes production traffic to shadow databases to simulate real-world loads without impacting live data.
Database Tools - Distributed SQL transaction and sharding engine.
High Availability and Scaling - Distributed SQL engine for sharding, scaling, and encryption.

dolthub/dolt

23,592View on GitHub

Dolt is a relational database engine that integrates version control directly into the database management layer. It functions as a version-controlled SQL database that tracks every row and schema change using a commit-based history, allowing users to branch, merge, and audit data modifications. By implementing a wire-protocol-compatible server, the system enables standard SQL clients and tools to interact with versioned data as if they were connecting to a traditional relational database. The platform distinguishes itself by applying repository-style workflows to data management, including s

redis/go-redis

22,159View on GitHub

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

doocs/advanced-java

78,987View on GitHub

This project is a comprehensive Java backend engineering guide and technical reference focused on high-concurrency design, distributed systems, and microservices architecture. It provides detailed strategies for decomposing monolithic applications, managing service discovery, and implementing the architectural patterns required for scalable backend environments. The repository distinguishes itself through an extensive collection of big data algorithmic references and database scaling strategies. It covers memory-efficient techniques for analyzing massive datasets, such as Top-K element extrac

vitessio/vitess

20,788View on GitHub

Vitess is a database clustering system for horizontal scaling of MySQL. It functions as a middleware layer that abstracts complex sharding and physical topology, allowing applications to interact with a distributed database environment through a unified interface. By intercepting and routing SQL queries across multiple shards, it enables large-scale data management while maintaining the appearance of a single database instance. The platform distinguishes itself through its ability to perform online schema migrations and distributed transaction coordination without requiring application downti

apacheshardingsphere

Features