39 Repos
Mechanisms for offloading large media assets to remote object storage providers to decouple content from local disk capacity.
Distinct from Object Storage: Distinct from general object storage: focuses on offloading media assets for storage decoupling rather than general data persistence.
Explore 39 awesome GitHub repositories matching data & databases · Remote Object Storage Integrations. Refine with filters or upvote what's useful.
CVAT ist ein Open-Source-Annotationstool für Computer Vision und eine Plattform zur Verwaltung visueller Datensätze. Es bietet eine selbst gehostete Schnittstelle zum Labeln von Bildern, Videos und 3D-Daten, um Datensätze für Vision-KI-Modelle zu erstellen. Die Plattform bietet KI-gestützte Daten-Labeling-Funktionen zur Automatisierung der Erstellung von Masken und Bounding Boxes und nutzt ein Plug-in-System zur Anbindung externer Modelle für maschinelles Lernen. Sie enthält ein konsensbasiertes Qualitätssicherungssystem, das die Genauigkeit von Labels durch den Vergleich unabhängiger Annotationen überprüft. Das System deckt kollaboratives Teammanagement, Projektorganisation durch Aufgabenzerlegung und die Integration von Remote-Cloud-Speichern ab. Es bietet zudem eine REST-API für die programmatische Workflow-Steuerung sowie den Import und Export von Daten in branchenüblichen Formaten.
Connects to cloud buckets and external file systems to stream large datasets without requiring full local storage.
PeerTube is a decentralized, open-source video hosting platform that enables users to operate independent, interoperable servers. By utilizing the ActivityPub protocol, it connects these servers into a global, federated network where users can follow channels, discover content, and interact across different instances. The platform is designed to function as a self-hosted video content management system, providing a community-driven alternative to centralized media services. What distinguishes PeerTube is its hybrid approach to content delivery and infrastructure management. It integrates peer
Supports remote object storage to decouple media assets from local disk capacity.
Owncast is a self-hosted live streaming server that provides full control over broadcast infrastructure and audience data. It functions as an RTMP video streaming server, accepting incoming video feeds and distributing them to viewers through HLS-based segmented streaming. The platform includes a built-in, stateful web-based chat interface that enables real-time viewer engagement during broadcasts. The project distinguishes itself through deep integration with the decentralized Fediverse, allowing servers to automatically broadcast stream status updates and notify followers across distributed
Offloads media segments to external cloud storage providers to reduce local bandwidth and storage requirements.
s3fs-fuse is a FUSE-based filesystem that mounts Amazon S3 buckets and S3-compatible object storage as local directories. It functions as a cloud object storage gateway, translating standard POSIX filesystem calls into RESTful API requests to allow remote objects to be accessed as if they were local files. The project enables the mapping of remote object stores to local paths on Linux, macOS, and FreeBSD. It supports integration with third-party S3-compatible providers through the use of custom service URLs and specialized request styles. The system includes capabilities for local data and m
Caches remote object storage data and metadata on local disks to improve performance.
RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen
Caches data on local disks to minimize the frequency and cost of requests to remote object storage.
MWPhotoBrowser is an iOS photo gallery component and media selection tool. It provides a grid-based browser for displaying photos and videos sourced from local storage, system libraries, or web URLs. The component includes a metadata captioning overlay that maps external text properties to descriptive captions displayed over media. To optimize the browsing experience, it utilizes a disk-based remote media cache that saves web images and videos locally to reduce network requests. The system supports a media selection workflow allowing users to pick multiple items using checkboxes and state-ba
Implements local disk caching of remote images and videos to reduce network requests during scrolling.
MLOps-Basics is a collection of implementation guides and blueprints for automating the machine learning lifecycle. It provides practical workflows for managing the transition of models from training to production deployment, focusing on the integration of operational tools into the machine learning pipeline. The project features specific architectural patterns for deploying containerized models using serverless infrastructure and cloud registries. It includes frameworks for tracking large datasets and model artifacts via remote storage, as well as guides for converting models into standardiz
Configures integrations with remote object storage to provide reproducible access to large-scale datasets.
This is a high-performance image component and native caching library for mobile applications. It provides tools for image optimization and asset preloading to reduce visual flicker and increase loading speeds. The project differentiates itself through a mobile asset preloader that downloads remote images into a local cache before they are requested. It also includes a system for assigning loading priority levels to ensure critical visual assets are fetched first and supports remote asset authentication by attaching custom HTTP headers to requests. The library covers a broad range of capabil
Implements local disk caching to store remote images on the device file system, reducing network requests.
Fine Uploader is a browser file upload widget and manager that provides a frontend interface for transferring multiple files. It functions as a chunked file upload manager and a client-side image processor. The project enables the direct transfer of files to cloud storage providers, specifically Amazon S3 and Microsoft Azure, to reduce the load on application servers. It includes tools for scaling and resizing image dimensions during the upload process to save bandwidth. The system manages large file transfers by splitting them into small pieces, allowing for pause and resume functionality.
Offloads large media assets directly to remote object storage providers to decouple content from the server.
Dieses Projekt bietet eine containerisierte DevOps-Plattform durch das Paketieren einer vollständigen GitLab-Installation in Docker-Images. Es ermöglicht die Bereitstellung einer selbst gehosteten Umgebung, die Git-Versionskontrolle, Projektmanagement sowie Continuous-Integration- und Delivery-Pipelines auf privater Infrastruktur integriert. Die Implementierung unterstützt die Bereitstellung via Docker Compose oder Orchestrierung durch Docker Swarm, was skalierbare Stacks mit integrierten Container-Registries ermöglicht. Sie nutzt Umgebungsvariablen für die Konfiguration und unterstützt das Auslagern von Artefakten und Backups auf entfernte Objektspeicher und externe Datenbank-Container. Die Plattform deckt ein breites Spektrum an operativen Funktionen ab, einschließlich Identitätsintegration mit LDAP, SAML und OAuth sowie automatisierter System-Backup- und Wiederherstellungsroutinen. Sie enthält zudem Werkzeuge für sicheres Networking via SSL/TLS-Konfiguration, Proxy-Traffic-Routing und das Hosten statischer Projektseiten. Administrative Aufgaben werden über Befehlszeilen-Dienstprogramme für Wartung und Gesundheitsüberwachung verwaltet, während die Systemkommunikation über SMTP- und IMAP-Konfigurationen abgewickelt wird.
Offloads large artifacts and files to remote object storage to reduce local disk usage.
This project is a comprehensive educational resource and curriculum focused on site reliability engineering, distributed systems, and infrastructure operations. It provides technical guides, a systems engineering course, and instructional manuals designed to teach the principles of managing large-scale computing environments. The curriculum covers high-level architectural design for scalability and resilience, including fault-tolerant infrastructure, high-availability patterns, and microservices decomposition. It emphasizes the practical application of site reliability engineering through the
Explains mechanisms for offloading static media assets to remote object storage providers.
Vendure is a Node.js e-commerce engine and headless commerce framework built with NestJS and TypeScript. It serves as a multi-channel commerce platform that manages product catalogs, orders, and customers via a strongly typed GraphQL API. The platform is distinguished by its highly extensible architecture, featuring a customizable administrative dashboard where developers can inject custom React components and entity views. It supports multi-channel commerce, allowing the isolation of products, currencies, and regional catalogs from a single unified backend. The engine covers a broad range o
Integrates with external object storage providers like AWS S3 and MinIO to host media assets.
Sui is a blockchain platform featuring an object-centric state model and resource-oriented smart contracts. It utilizes parallel transaction execution to increase network throughput and supports programmable transaction blocks that bundle multiple operations into single atomic units. The platform distinguishes itself with a capability-based access control system and zero-knowledge login mechanisms, enabling users to authenticate via identity providers without seed phrases. It also implements deterministic object addressing to allow predictable state lookups and supports the creation of soulbo
Writes compressed protobuf-encoded checkpoint blobs to remote object stores like S3 or GCS.
The Kaggle API command line interface is a suite of utilities for managing datasets, machine learning models, and competition entries from a terminal. It functions as a command line wrapper that translates user input into API calls to control remote cloud resources. The project differentiates itself by providing specialized tools for automating the execution of notebook kernels and managing the lifecycle of machine learning models, including version iteration and performance tracking. It also includes a utility for executing evaluation tasks against large language models and downloading the r
Caches downloaded datasets and models on the local disk to prevent redundant network transfers.
HedgeDoc is a self-hosted documentation platform and real-time collaborative Markdown editor. It serves as a digital workspace for creating shared technical notes and managing knowledge through a privately hosted system. The platform enables multiple users to write and format Markdown documents simultaneously in a shared live environment. It integrates external identity providers such as LDAP, SAML, and social platforms via OAuth2 to manage user access and authentication. The system includes capabilities for content publishing, including exporting notes to GitHub Gists and generating automat
Provides mechanisms for offloading media assets to remote object storage providers to decouple content from local disk.
OS.js is a browser-based desktop environment built entirely with JavaScript that runs a full graphical interface with windows, file management, and applications inside any modern web browser without requiring local software installation. It provides a web desktop platform that manages application windows as DOM elements with z-ordering, resizing, and event routing, while offering an abstracted virtual filesystem layer that unifies access across multiple storage backends through a common read/write interface. The platform connects to remote servers and desktops using SSH, VNC, or Xpra protocol
Reads and writes files through a unified filesystem layer that works across different backends including local, Google Drive, WebDAV, and FTP.
Boto is a Python SDK and API wrapper for Amazon Web Services. It serves as a programmatic interface for managing and automating cloud infrastructure, mapping cloud-side resources to native Python objects and methods. The library provides tools for the programmatic control and orchestration of compute, storage, networking, and database resources. It enables the automation of infrastructure deployments and the management of virtual servers, container services, and serverless functions. Capability areas include identity and access management, cloud monitoring and observability, and the administ
Manages the storage and retrieval of unstructured data and long-term archives using S3-compatible services.
Overtone ist eine Lisp-Bibliothek für Audiosynthese, die eine programmierbare Umgebung für das Design von Synthesizern und die Generierung von Echtzeit-Audio bietet. Sie dient als Framework für digitale Signalverarbeitung (DSP) im Live-Coding-Bereich und integriert einen Audio-Sample-Manager zum Abrufen und Caching von Sounds aus Remote-Repositories sowie einen Musiksequenzer für die Komposition melodischer und rhythmischer Strukturen. Das Projekt zeichnet sich durch spezialisierte Schnittstellen für Hardware und Networking aus, darunter ein dediziertes System für das Routing von MIDI-Daten und den Austausch von Open Sound Control-Nachrichten zwischen Software und externer Hardware. Es enthält zudem Tools für algorithmische Musikkomposition, wie die Generierung euklidischer Rhythmen sowie Funktionen zur Definition von Tonleitern und Akkorden. Das Framework deckt ein breites Funktionsspektrum ab, einschließlich digitaler Audiosignalverarbeitung mit Node-basiertem Routing und individuellem Envelope-Design. Es bietet Systeme für die Synchronisation von Audio-Events, zeitgesteuertes Sequencing sowie die Möglichkeit, Audio-Output mit Grafikbibliotheken und Shader-Umgebungen für audiovisuelle Synchronisation zu koordinieren. Zusätzliche Utilities umfassen Tonhöhen- und Akkordanalysen sowie Echtzeit-Monitoring von Audio- und Control-Bus-Werten.
Caches remote audio samples on local disks to reduce retrieval latency and network requests.
The AWS SDK for PHP is a software development kit that provides HTTP client classes for every supported Amazon Web Service, enabling PHP applications to send authenticated requests and receive structured, typed response objects. It includes a credential resolution chain that automatically locates credentials from environment variables, instance profiles, or configuration files, and supports promise-based asynchronous execution for running multiple API calls concurrently to improve throughput. The SDK distinguishes itself through a middleware pipeline architecture that allows interception and
Provides PHP stream wrappers and multipart uploads for managing S3 objects like local files.
GVFS is a virtual file system and performance optimizer designed to manage enterprise-scale Git repositories. It provides a virtualization layer that allows large remote repositories to be treated as local directories by retrieving file contents from a server only when they are accessed. The system optimizes version control operations by limiting the scope of active file objects and downloading specific data on demand rather than cloning the entire repository history locally. This approach reduces initial setup time and minimizes local disk usage. The project implements a workflow based on o
Implements a local disk cache for retrieved file fragments to minimize redundant network requests.