Electric

Electric is a Postgres data synchronization engine and replication proxy designed to enable local-first software. It replicates data from Postgres databases to client-side stores in real time using logical replication, allowing applications to maintain a local embedded database for offline access and low-latency updates.

The system distinguishes itself by using shapes to filter and authorize specific subsets of database rows and columns before streaming them to clients or edge workers. It further supports multi-user collaboration by integrating a conflict-free replicated data type framework to ensure consistent state synchronization across different users.

The project covers a broad range of capabilities, including reactive state management and real-time data streaming to client interfaces and server-side renders. It provides tools for data shaping and transformation, database integration across various cloud and serverless Postgres providers, and security primitives such as token-based authorization and end-to-end encryption.

The service can be deployed as a containerized web service on cloud platforms with support for rolling deployment management.

Features

Data Synchronization Engines - Connects to Postgres databases using logical replication to synchronize data across clients in real time.
Local-First Architectures - Implements a local-first architecture by synchronizing remote Postgres data to embedded client databases for offline access.
Client-Side State Synchronizers - Synchronizes remote Postgres database data into local client-side stores for low-latency web and mobile access.
Change Data Capture Streams - Consumes a real-time log of database changes into a stream to materialize local data structures.
Data Shaping - Allows defining specific rows and columns from tables to be synchronized using filters and selections.
Dynamic Sync Shapes - Allows the creation of personalized data structure definitions at runtime based on the current session and request.
Database Replication Middleware - Configures how database tables publish for replication using automated or manual privilege control modes.
Remote Synchronization - Implements background synchronization of data from an embedded local database to a remote server.
Client-Side Materializers - Streams database changes to a JavaScript environment using an interface that materializes change logs into local state.
Log Materialization - Transforms a stream of logical database operations into a local data structure to maintain current state.
Real-time Data Synchronization - Streams database changes from Postgres to client applications to update user interfaces instantly.
Shape-Based Synchronization - Defines data shapes at compile time to synchronize specific database subsets in real time using authentication middleware.
Live Update Subscriptions - Pushes database changes to clients via persistent connections to reduce latency and request overhead.
Local-First Synchronization - Provides a synchronization layer that streams database changes to embedded client stores for offline-first functionality.
Database Change Streaming - Streams specific subsets of Postgres database rows and columns to clients or edge functions via logical replication.
Conflict-Free Replicated Data Types - Integrates a conflict-free replicated data type (CRDT) framework with a database to enable consistent multi-user collaboration.
Real-Time Data Streaming - Offers an API that pushes live database updates to connected clients via persistent connections.
Real-time Collaboration - Transports conflict-free replicated data types (CRDTs) to support multi-user editing and shared application state.
Incremental Sync - Provides a mode to omit the initial data snapshot and stream only new changes after connection.
Database Integrations - Integrates with cloud-hosted Postgres instances via connection strings to synchronize data with clients.
Shard Routing Logic - Directs client connections to specific database shards using client-side logic or edge routing to distribute load.
Edge-to-Cloud Synchronization - Hydrates data into edge workers or durable objects to enable low-latency processing at the network edge.
On-Demand Subset Loading - Reduces initial load times by loading specific data subsets on-demand via snapshots or query-driven synchronization.
Postgres Adapters - Establishes direct connections to serverless Postgres hosts to enable logical replication.
Real-Time Data Streaming - Pushes real-time database updates into serverless edge functions to generate dynamic responses.
Mobile Synchronization - Synchronizes data from a remote database into mobile applications to provide real-time updates and offline access.
Synchronization Snapshot Optimizations - Optimizes data transfer by selecting between full historical snapshots or streaming only new changes for initial synchronization.
API Access Security - Requires secret tokens for all HTTP API calls to prevent unauthorized access to the synchronization service.
API Authentication - Secures data access using gatekeepers or proxies to authorize requests between the client and synchronization engine.
End-to-End Encryption - Protects sensitive information by encrypting data at the client before transmission and decrypting it upon receipt.
Reverse Proxy Security - Validates user credentials and shape parameters through a reverse proxy before streaming data to clients.
Token-based Authorization - Issues shape-scoped tokens that are validated by a proxy to ensure only authorized data subsets are requested.
Component Data Bindings - Binds materialized data shapes to component state using hooks to manage the subscription lifecycle.
Reactive Data Stores - Synchronizes database state to a local reactive store that automatically triggers UI updates.
User-to-Shard Migrations - Moves users between shards by updating mapping and triggering a client-side resync via control messages.
Reactive Data Bindings - Triggers callbacks or broadcasts when data changes to automatically update the user interface.
Collaborative Editing Frameworks - Combines real-time synchronization with conflict-resolution frameworks to enable multi-user collaborative editing.
Reactive State Management - Binds database change streams to client-side stores to automatically trigger UI updates.
Observable State Patterns - Streams database changes into an observable state container to enable automatic user interface updates.
Real-Time Collaboration Tools - Provides a framework for building multi-user applications with shared state and real-time synchronization.
Sync-Aware Rendering - Integrates real-time data synchronization with server-side rendering to ensure seamless transitions to client-side updates.

pubkey/rxdb

23,048View on GitHub

This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

realm/realm-swift

16,608View on GitHub

This is a mobile object database and NoSQL local data store that replaces relational tables with a schema-based model. It functions as a reactive data store, using live object observations and change notifications to trigger automatic user interface refreshes. The system provides built-in mobile cloud data synchronization to keep local datasets consistent with a remote server across multiple devices. It also includes security features for encrypted local storage, protecting sensitive on-disk data using at-rest encryption keys and fine-grained access control. Broad capabilities include object

yjs/yjs

22,069View on GitHub

Yjs is a CRDT framework and real-time state synchronization engine providing shared data types for collaborative software. It functions as a shared data type library of collaborative maps, arrays, and text structures that converge automatically across distributed clients, serving as a local-first data store and a collaborative editor backend. The system distinguishes itself through a synchronization engine that encodes document changes into commutative binary updates. It employs state-vector based differencing to transmit only missing incremental updates between peers and uses relative-positi

hazelcast/hazelcast

6,570View on GitHub

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

electric-sqlelectric

Features