9 Repos
Systems ensuring the consistency, safety, and historical recoverability of data during write and synchronization operations.
Explore 9 awesome GitHub repositories matching data & databases · Data Integrity and Versioning. Refine with filters or upvote what's useful.
Syncthing ist eine dezentrale Dateisynchronisierungs-Engine, die konsistente Datenzustände über mehrere Geräte hinweg durch Peer-to-Peer-Mesh-Netzwerke aufrechterhält. Es arbeitet als Hintergrund-Daemon, der Dateierstellungen, -änderungen und -löschungen automatisch zwischen vertrauenswürdigen Knoten repliziert, ohne dass zentrale Server erforderlich sind. Durch die Verwendung von inhaltsadressierbarer Blockindizierung und Block-Level-Delta-Synchronisation identifiziert und überträgt das System nur die geänderten Segmente von Dateien, was eine effiziente Datenverbreitung in heterogenen Umgebungen gewährleistet. Das Projekt zeichnet sich durch eine sicherheitsorientierte Architektur aus, die auf gegenseitiger TLS-Authentifizierung basiert, um die Geräteidentität zu verifizieren und sicherzustellen, dass alle Verbindungen kryptografisch an vertrauenswürdige Zertifikats-Fingerabdrücke gebunden sind. Es unterstützt flexible Synchronisationsmodi, einschließlich bidirektionaler Replikation, unidirektionaler Spiegelung für Backups und referenzbasierter Durchsetzung. Für zusätzliche Privatsphäre bietet das System eine ordnerbasierte Verschlüsselung für nicht vertrauenswürdige Geräte und ermöglicht eine granulare Kontrolle über den Netzwerkverkehr, einschließlich der Möglichkeit, Vorgänge auf lokale Netzwerke zu beschränken oder Relay-Infrastruktur für NAT-Traversal zu nutzen. Über die Kernfunktionen der Replikation hinaus bietet die Plattform umfassende Verwaltungstools, darunter ein webbasiertes Dashboard zur Überwachung von Verbindungsstatus und Durchsatz sowie eine Befehlszeilenschnittstelle für fortgeschrittene Konfigurationen. Es enthält robuste Versionierungsstrategien zum Schutz vor Datenverlust und unterstützt komplexe Bereitstellungsszenarien durch native Service-Integration und Observability-Metriken. Die Software ist für plattformübergreifende Kompatibilität ausgelegt und kann über Standard-Paketmanager oder containerisierte Umgebungen installiert werden.
Tracks data state by indexing block-level cryptographic hashes to identify differences and reconcile content between devices.
This project provides a comprehensive framework for building, training, and managing autonomous agents. It enables the construction of systems that utilize language models to plan, manage memory, and execute multi-step tasks through iterative reasoning loops and tool-based actions. The framework distinguishes itself by offering specialized capabilities for interacting with graphical user interfaces and legacy software, allowing agents to perceive visual elements and perform actions like a human user. It supports complex, cross-application workflows through graph-based orchestration and provid
Ensures batch file modifications are applied as atomic transactions to maintain system consistency.
This project is a comprehensive, community-maintained knowledge base and toolkit designed for competitive programming. It serves as a centralized repository for algorithmic theory, data structures, and mathematical techniques, providing a structured reference for informatics and collegiate programming competitions. The project distinguishes itself by integrating educational content with a robust suite of automation utilities. It provides a complete workflow for competitive programming, including tools for automated test case generation, solution verification, and direct interaction with onlin
Creates temporary backups of open files to prevent data loss.
Lowdb is a lightweight local data persistence library that manages application state as a JSON document. It functions as a database utility that stores information directly in local files or browser storage, removing the requirement for a dedicated database server or complex infrastructure. By treating stored data as standard objects and arrays, it provides a straightforward interface for managing application information. The library distinguishes itself through a modular adapter-based architecture that decouples data manipulation from the underlying storage medium. This design allows for the
Protects data integrity by performing write operations as single atomic transactions.
Seafile is a self-hosted file synchronization and sharing platform that provides a central server for maintaining file consistency across multiple devices. It functions as a cloud storage management system and a collaborative document suite, integrating tools for real-time teamwork and shared file management. The platform distinguishes itself through a metadata-driven file organizer that uses extensible properties and hierarchical tags instead of traditional folder structures. It includes client-side encrypted storage to protect private data using user-defined passwords before files leave the
Implements a versioning system that tracks file iterations to allow recovery of previous document states.
all-in-one is a containerized deployment system designed to install and manage a complete suite of productivity and collaboration services. It functions as a cloud suite deployer that orchestrates the installation of a self-hosted content platform, incorporating necessary dependencies via Docker or Kubernetes. The project distinguishes itself by providing a web-based dashboard for orchestrating, updating, and monitoring the lifecycle of service containers. It also serves as a local AI inference server, enabling the execution of generative text models, image diffusion, and speech processing on
Tracks and manages multiple iterations of files with decay schedules to allow recovery.
CarrierWave is a Ruby file upload library used to manage the uploading, storing, and retrieval of files within web frameworks such as Rails and Sinatra. It functions as an Active Record file manager that associates uploaded assets with database records. The project includes an image processing pipeline for generating thumbnails and derivative versions of uploaded images. It also features a file validation engine to restrict uploads based on allowlists or denylists of extensions and content types, and provides cloud storage integration to manage assets on remote providers. The library covers
Creates multiple derivative versions of a single uploaded file, such as thumbnails with specific dimensions.
TagSpaces is an offline-first file tagging and organization platform that lets you manage local files with portable metadata stored directly in filenames or sidecar JSON files, eliminating the need for a central database. It functions as a full-text file search engine, a Kanban board file organizer, a local AI file assistant, an S3-compatible cloud file manager, and a web clipper and bookmark manager, all within a single application. The project distinguishes itself through a local-first architecture where all file operations, indexing, and AI processing run entirely on the device, with cloud
Keeps previous versions of edited files to allow restoration and prevent data loss.
Syncthing macOS is a native application bundle that wraps the Syncthing binary to provide decentralized, peer-to-peer file synchronization on macOS. It enables bidirectional data replication across multiple devices using a block exchange protocol, allowing files and folders to stay consistent without relying on a central cloud server. The project distinguishes itself by integrating the cross-platform synchronization engine into the macOS environment as a background service. It provides a system tray interface for quick access to the management GUI and supports automatic application startup th
Compares local file states against global versions and transfers missing data blocks for consistency.