10 Repos
Tools for transferring data between different persistence backends.
Distinct from Pluggable Storage Backends: Candidates focus on configuration and adapters, but not the actual process of migrating data between backends.
Explore 10 awesome GitHub repositories matching data & databases · Storage Migration. Refine with filters or upvote what's useful.
kops is a Kubernetes cluster provisioner and lifecycle manager designed to automate the creation, maintenance, and destruction of production-grade clusters on cloud infrastructure. It functions as a declarative infrastructure manager, synchronizing the live state of a cluster with versioned manifests stored in remote object storage to ensure idempotent operations. The project distinguishes itself by offering comprehensive automation for the entire cluster lifecycle, including high-availability control plane deployment, incremental rolling updates, and automated version upgrades. It also serve
Moves cluster configuration and state files between different cloud storage buckets or providers.
tdl is a command-line tool for Telegram account automation, media management, and data archiving. It provides a programmatic interface for downloading and uploading files, forwarding messages, and exporting chat history, member lists, and media into structured JSON files. The project distinguishes itself through a session management system that isolates multiple account identities using unique namespaces. It features expression-based routing for messages and uploads, allowing users to direct content to specific destination chats using custom logic and dynamic caption generation. The tool cov
Transfers existing stored data from one storage backend to another.
Cronicle is a distributed job scheduler that replaces traditional cron with a browser-based management interface. It runs scheduled tasks across a cluster of servers with automatic failover, using a custom cron parser that intersects day-of-month and day-of-week constraints when both are specified. The system executes jobs through a plugin framework that runs command-line scripts in any language, communicating via JSON over standard input and output. The scheduler provides a web-based real-time dashboard for monitoring running jobs with live logs, resource usage charts, and progress updates.
Reads all storage records from the current location and writes them to a new location or engine.
Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro
Switches a cluster's storage backend from chunks to blocks by gradually replacing ingesters.
Kvrocks is a disk-based NoSQL database and distributed key-value store that leverages the RocksDB storage engine to persist large datasets to physical disk. It is designed to be a Redis-compatible database, utilizing the standard Redis communication protocol to ensure interoperability with existing client libraries and tools. The project distinguishes itself by combining a disk-persistent storage model with advanced retrieval capabilities, including vector search for k-nearest neighbor queries, full-text search indexing, and geospatial query execution. It supports distributed clustering with
Moves datasets between memory-resident and disk-based storage backends.
wal-g ist ein Datenbank-Cloud-Backup-Tool und Write-Ahead-Log-Archiver für Disaster Recovery und Point-in-Time-Recovery. Es archiviert Datenbankstatus und Transaktionslogs bei Cloud-Speicheranbietern, um langfristige Datenhaltbarkeit sicherzustellen. Das Projekt bietet einen Manager zum Komprimieren, Verschlüsseln und Bereinigen von Datenbank-Backups, die in Cloud-Objektspeichern abgelegt sind. Es ermöglicht spezifisch die Wiederherstellung einer Datenbank auf einen bestimmten Zeitstempel durch Abruf von Basis-Backups und Replay von Transaktionslogs. Die Funktionen des Tools decken Multi-Engine-Backup und -Wiederherstellung ab, einschließlich der Generierung von Voll- und Delta-Backups. Es verwaltet den Backup-Lifecycle durch Inventar-Tracking und automatisierte Bereinigungsrichtlinien und enthält Utilities zur Überprüfung der Integrität von Log-Archiven und Tabellendateien. Ein lokaler Proxy ist verfügbar, um Cloud-Speicher-APIs für das Testen von Backup- und Restore-Operationen zu emulieren.
Moves or copies database backups and logs between different configured storage backends.
Ombi is a media request management system that coordinates content discovery between users, media servers, and download managers. It acts as a middleware layer to automate the process of requesting, approving, and acquiring movies, music, and TV shows. The project differentiates itself by acting as an integration layer that synchronizes libraries and user accounts across multiple media server instances. It features a content request orchestrator that supports voting-based automatic approval and the ability to import active requests directly from external watchlists. The platform covers a bro
Moves existing application data from a local SQLite file to a MySQL or MariaDB server.
Flipper ist ein Feature-Toggle-System und eine Library für Ruby-Anwendungen. Es ermöglicht die Verwaltung von Anwendungsfunktionen durch das Ein- und Ausschalten von Features, ohne dass neuer Code bereitgestellt werden muss. Das System bietet einen Mechanismus zur Steuerung der Codeausführung durch boolesche Zustände, akteur-basierten Zugriff und prozentuale Traffic-Rollouts. Es enthält eine Verwaltungsoberfläche zur Echtzeit-Änderung von Flag-Zuständen sowie eine zentrale Registry zur Nachverfolgung verfügbarer Funktionen. Das Projekt deckt ein breites Spektrum an operativen Möglichkeiten ab, darunter umgebungsspezifische Flag-Isolierung, Konfigurationssynchronisation über Deployment-Ziele hinweg und Audit-Logging zur Nachverfolgung von Änderungen. Es unterstützt verschiedene Storage-Backends über eine Adapter-Schicht und nutzt eine Caching-Strategie zur Reduzierung der Abfragelatenz. Zusätzliche Funktionen umfassen zeitgesteuerte Feature-Aktivierung, Canary-Release-Management und automatisierte Benachrichtigungen via Slack-Integration.
Provides utilities for copying feature configurations between different storage backends during infrastructure transitions.
Papra is a self-hosted document management system designed for digital archiving, organization, and retrieval. It serves as a centralized platform for storing files with a focus on security, providing an encrypted file archive using AES-256-GCM and a programmatic interface for managing documents and metadata via a REST API, SDK, and command line tools. The system distinguishes itself through an automated document ingestion engine that imports files via email forwarding, monitored folders, and webhook listeners. It further enhances discoverability by acting as an OCR document indexer, extracti
Provides tools to migrate documents between different storage backends while preserving encryption and security.
Filegator is a web-based file manager and self-hosted file sharing portal. It functions as a cloud storage gateway that allows users to organize, upload, and download files across local disks and various cloud storage providers through a single unified interface. The system features a multi-backend authentication provider capable of verifying identities via JSON files, databases, or external directory services. It includes a role-based access control system that manages user permissions, assigns home folders, and restricts access to specific files and system routes. The platform provides cap
Copies data from a source provider to a replica in real-time to transition between storage backends.