Awesome GitHub RepositoriesPersistent Storage Providers

Puter is a browser-based desktop environment and cloud-native development platform that provides a virtualized graphical workspace. It enables developers to build and deploy full-stack web applications by integrating cloud storage, authentication, and serverless backend logic directly into the browser, eliminating the need for traditional server infrastructure. The platform distinguishes itself through a unified cloud storage layer and a distributed network runtime that facilitates peer-to-peer communication and cross-origin resource fetching. It features a sophisticated cross-window orchestr

This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

Automa is a browser-based automation platform that enables users to build, schedule, and execute repetitive web tasks through a visual, no-code interface. By operating as a browser extension, it provides a canvas-based environment where users construct workflows by connecting functional blocks to interact with web elements, manage browser state, and process data. The platform distinguishes itself through its deep integration with the browser environment, allowing for complex orchestration such as event-driven triggers, cross-origin request handling, and the ability to package workflows as sta

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Memori is an AI agent memory middleware platform designed to provide persistent, context-aware recall for language models. It functions as a non-intrusive layer that intercepts outbound model requests to automatically capture interaction history and execution traces, ensuring that agents maintain continuity across sessions without requiring modifications to existing application logic. The platform distinguishes itself through a dual-model storage architecture that maintains information as both structured relational primitives for precise fact retrieval and rolling narrative summaries for situ

Annoy is a C++ library designed for approximate nearest neighbor search in high-dimensional vector spaces. It functions as a vector similarity search engine that constructs static, disk-based data structures to facilitate fast lookups. By mapping identifiers to vector data and persisting these structures to disk, the library enables efficient, memory-mapped access to large datasets. The project distinguishes itself through the use of random projection trees and distance-metric-based partitioning, which organize data into hierarchical binary trees to balance search precision against computatio

Dask ist ein Framework für paralleles Rechnen und ein verteilter Task-Scheduler, der darauf ausgelegt ist, Python-Data-Science-Workflows von einzelnen Maschinen auf große Cluster zu skalieren. Es fungiert als Cluster-Ressourcenmanager, der die Berechnungslogik orchestriert, indem Aufgaben und deren Abhängigkeiten als gerichtete azyklische Graphen dargestellt werden. Diese Architektur ermöglicht es dem System, die Verteilung von Workloads auf verfügbare Hardware zu automatisieren und gleichzeitig komplexe Ausführungsanforderungen zu verwalten. Das Projekt zeichnet sich durch eine Lazy-Evaluation-Engine aus, die Datenoperationen verzögert, bis sie explizit angefordert werden, was eine globale Graphoptimierung und effiziente Ressourcenzuweisung ermöglicht. Es integriert speicherbewusstes Data-Spilling, um Systemabstürze bei der Verarbeitung von Datensätzen zu verhindern, die den verfügbaren Speicher überschreiten, und nutzt Task-Graph-Fusion, um Sequenzen von Operationen in einzelne Ausführungsschritte zu kombinieren, wodurch Scheduling-Overhead und Inter-Node-Kommunikation minimiert werden. Die Plattform bietet eine umfassende Oberfläche für die Datenanalyse im großen Maßstab, einschließlich Unterstützung für verteiltes maschinelles Lernen, Integration in das Hochleistungsrechnen und parallele Datenverarbeitung. Sie bietet umfangreiche Werkzeuge für das Cluster-Lebenszyklusmanagement, Performance-Profiling und die Echtzeitüberwachung der Aufgabenausführung. Benutzer können diese Umgebungen über verschiedene Infrastrukturen hinweg bereitstellen, einschließlich lokaler Hardware, Cloud-Anbietern, containerisierten Systemen und Hochleistungsrechner-Clustern.

Redux Persist is a state management utility designed to maintain application data across sessions by synchronizing state with local storage. It functions as a middleware-based engine that automatically saves and restores application state, ensuring that user sessions remain intact after page refreshes or application restarts. The library distinguishes itself through granular control over the persistence lifecycle and data structure. It supports recursive state tree partitioning, allowing developers to apply distinct storage configurations and filtering rules to specific branches of the applic

OpenRefine is a data cleaning tool and wrangling platform used to transform raw, messy datasets into consistent and structured formats. It operates as a Java-based data processor that runs a local server and provides a web browser interface for managing and manipulating data. The platform includes a data reconciliation engine for matching local entries against external knowledge bases to standardize entities. It also functions as a web data augmentation tool, allowing users to fetch and integrate information from external web sources to enrich their datasets. The system provides a transforma

GPTCache is a semantic caching layer and response optimizer for large language models. It functions as pluggable middleware for orchestration frameworks, utilizing vector database caching to store and retrieve model responses based on the semantic similarity of prompts rather than exact text matches. The system uses embeddings to determine cache hits by comparing the distance between new queries and stored vectors. It employs a hybrid storage model that persists original prompts in relational databases while maintaining high-dimensional embeddings in vector stores. The project covers a broad

Riverpod is a reactive state management framework for Dart that manages application state through providers which automatically track dependencies and rebuild dependents when values change. It functions as an asynchronous data caching library, caching results of async operations and exposing loading, error, and data states to widgets without manual future handling. The framework includes a provider composition system that combines multiple providers into derived values that recompute automatically when any dependency changes, along with code generation for state providers that reduces boilerpl

FASTER is a high-throughput key-value store that combines an in-memory data store with a hybrid memory-disk storage engine, enabling datasets larger than available RAM. It uses a latch-free, cache-optimized index for concurrent point lookups and heavy updates, and records all mutations to a persistent append-only log on disk with checksum validation and group-commit checkpointing for crash recovery. The system supports multi-key transactional workloads through atomic multi-key locking, ensuring transactional consistency without coarse-grained contention. It exposes the key-value store to remo

Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c

PikiwiDB ist eine verteilte NoSQL-Datenbank und ein disk-basierter Key-Value-Store, der als Redis-kompatibler Protokollserver fungiert. Er ist darauf ausgelegt, Datensätze zu verarbeiten, die größer als der verfügbare Arbeitsspeicher sind, indem eine Persistenz-Engine genutzt wird, die den vollständigen Datensatz auf der Festplatte speichert. Das System verwendet ein Tiered-Storage-Modell, das häufig abgerufene Hot-Daten im Arbeitsspeicher zwischenspeichert, während das primäre Volumen auf der Festplatte verbleibt. Es stellt eine hohe Verfügbarkeit durch eine replizierte Datenspeicherarchitektur sicher und verwendet asynchrone Binär-Logs, um Daten zwischen primären und sekundären Knoten zu synchronisieren. Das Projekt unterstützt die Skalierung verteilter Datenbanken durch cluster-basiertes Data-Sharding und organisiert Daten in Gruppen, um die Speicherschicht zu erweitern. Zu den operativen Funktionen gehören System-Performance-Monitoring zur Verfolgung der Ressourcennutzung sowie die Unterstützung für containerisiertes Deployment.

SparkInternals ist ein technisches Referenz- und Architekturhandbuch, das das interne Design und die Implementierung der verteilten Computing-Engine Apache Spark detailliert beschreibt. Es dient als Analyse von Big-Data-Engines und konzentriert sich darauf, wie das System die Cluster-Ausführung sowie das Zusammenspiel zwischen Driver-Nodes, Executors und Workern verwaltet. Das Projekt bietet eine detaillierte Aufschlüsselung, wie logische Pläne in physische Ausführungsstufen konvertiert werden. Es analysiert spezifisch die Mechanik von Data-Shuffle-Operationen, Speicherverwaltung und die Koordination der verteilten Job-Planung. Die Dokumentation deckt ein breites Spektrum an verteilten Computing-Funktionen ab, einschließlich Query-Execution-Planung, Datenabhängigkeitsmanagement und In-Memory-Caching-Strategien. Zudem werden Aufgabenverteilung, parallele Ausführung sowie Prozesse zur Fehlerwiederherstellung und Datenpersistenz untersucht.

This project is a .NET identity stack and server framework used to build certified OpenID Connect and OAuth 2.0 identity providers. It provides the core logic required to issue and validate security tokens and manage user authentication across various grant types and protocol flows. The framework includes a protocol translation layer that bridges OpenID Connect and SAML to enable interoperability between different identity providers. It also supports a stateless mode of operation, which removes built-in validation and storage to allow for manual control over token and client verification. Th

android-common is a collection of shared utility components and framework libraries for Android development. It provides specialized toolkits for reverse engineering, system utility management, data caching, and high-performance user interface components. The project includes a reverse engineering toolkit for inspecting application internals through package decompilation and manifest data extraction. It also features a system utility toolkit for managing file operations and executing shell commands within the Android operating system. The library covers several capability areas, including da

MiniOB is an open-source educational relational database kernel designed for learning the internals of database systems. It implements a dual-engine storage architecture combining B+ Tree and LSM-Tree, supports SQL parsing and query execution, and provides transactional processing with multi-version concurrency control. The system communicates with clients using the MySQL wire protocol and includes a vector database extension for storing and querying high-dimensional vectors. The project distinguishes itself through its comprehensive coverage of core database concepts in a single, learnable c