Awesome GitHub RepositoriesCloud-Native Storage Layers

MinIO is a software-defined, cloud-native object storage server designed to manage large volumes of unstructured data. It functions as a distributed storage cluster that aggregates multiple independent nodes into a unified, scalable pool, providing a high-performance infrastructure compatible with standard cloud storage protocols and application programming interfaces. The system utilizes a shared-nothing architecture that eliminates central metadata servers, relying instead on a decentralized hash table to map objects across the cluster. Data availability and resilience are maintained throug

go-ipfs is an implementation of an IPFS node, providing a distributed filesystem and a content-addressable storage system. It enables the storage and retrieval of data based on unique cryptographic hashes rather than fixed network locations, allowing files to be shared across a peer-to-peer network without a central authority. The system utilizes a distributed hash table and a peer-to-peer gossip protocol to route requests and propagate network state and metadata. It organizes data using a Merkle DAG structure to support efficient deduplication and versioning of content. Capabilities include

CVAT is an open-source, web-based platform designed for annotating images, videos, and 3D point clouds to create high-quality training datasets for machine learning. It functions as a containerized server that orchestrates the entire lifecycle of computer vision data, from initial task creation and manual labeling to quality assurance and final dataset export. The platform distinguishes itself through deep integration with machine learning models, allowing users to deploy custom AI models as serverless functions for automated object detection, tracking, and skeleton annotation. It supports co

JuiceFS is a distributed file system designed to mount object storage as a local, POSIX-compliant drive. It functions as a cloud-native persistent storage layer that decouples file metadata from raw data, storing metadata in a transactional database while keeping data blocks in object storage. This architecture enables multiple hosts across different regions to access the same storage simultaneously while maintaining strong consistency. The system distinguishes itself by performing data processing, including compression and encryption, directly on the client side before transmission. By split

SkyPilot is a multi-cloud AI orchestrator and distributed task scheduler designed to launch and manage AI workloads across various cloud providers, Kubernetes, and Slurm clusters. It functions as an infrastructure-as-code framework that uses declarative files to define resource requirements and setup commands for consistent execution across different environments. The project differentiates itself through automated cost optimization, selecting the most affordable GPU or TPU hardware and managing spot instances to reduce expenses. It also provides a remote development environment that bridges

KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models on personal hardware. It functions as a multimodal AI server and API gateway, providing OpenAI-compatible endpoints that allow third-party clients to interact with locally hosted models. The project distinguishes itself as an AI storytelling backend, featuring dedicated tools for long-form narrative management through persistent memory, world lore tracking, and character state management. It further extends its capabilities as a multimodal server capable of processing text, im

all-in-one is a containerized deployment system designed to install and manage a complete suite of productivity and collaboration services. It functions as a cloud suite deployer that orchestrates the installation of a self-hosted content platform, incorporating necessary dependencies via Docker or Kubernetes. The project distinguishes itself by providing a web-based dashboard for orchestrating, updating, and monitoring the lifecycle of service containers. It also serves as a local AI inference server, enabling the execution of generative text models, image diffusion, and speech processing on

h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

This is an open-source educational website that translates and localizes MIT's Missing Semester course, teaching practical computing skills for computer science students. The curriculum covers developer tooling, shell scripting, version control, security fundamentals, and open-source collaboration, with a focus on core computing skills including data processing pipelines, workflow automation, secure remote access, shell productivity, Vim editing, and Git version control. The project distinguishes itself by teaching command-line mastery, shell scripting, and automation to boost daily developer

ZeroByte is a backup management platform built around the Restic backup engine, providing encrypted, deduplicated, and compressed snapshots across multiple storage backends. It offers a web interface for scheduling, monitoring, and managing backup operations, with support for cron-based job scheduling and configurable retention policies that automatically prune older snapshots. The platform distinguishes itself through comprehensive multi-protocol volume mounting, allowing backup ingestion from NFS, SMB, WebDAV, SFTP, and rclone-backed sources alongside local directories. It includes a snapsh

SSHFS-Win is a Windows implementation of SSHFS that mounts remote directories over SSH as local Windows drives, enabling seamless file access as if they were local network drives. It provides both command-line and graphical interfaces for creating, managing, and disconnecting SSHFS mounts, supporting password or SSH key authentication with optional credential storage in the Windows Credential Manager. The project extends beyond basic SSH mounting to support a wide range of remote file access scenarios, including mounting cloud storage services like Azure Blob or Amazon S3, distributed POSIX f

Ce projet est un optimiseur de performance et un benchmark de ressources pour AWS Lambda. Il analyse le compromis entre la vitesse d'exécution et le coût en testant diverses configurations de mémoire pour identifier les paramètres les plus rentables et minimiser les dépenses opérationnelles. L'outil utilise un orchestrateur AWS Step Functions pour automatiser l'exécution et la collecte de données de plusieurs tests de fonctions à différents niveaux de puissance. Il simule des charges de travail de production en injectant des données statiques ou distantes personnalisées et en utilisant une distribution de charge utile pondérée pour imiter des modèles de trafic réels. La suite couvre plusieurs domaines de capacité, incluant l'échantillonnage itératif de la mémoire et la modélisation des coûts basée sur les métriques pour visualiser les compromis de performance. Elle fournit un nettoyage automatisé des ressources pour les versions et alias de fonctions temporaires, une configuration réseau privée pour les ressources internes restreintes et le chargement de charge utile à distance pour contourner les limites de taille d'invocation standard. Le déploiement est géré via des constructions d'infrastructure-as-code pour garantir une configuration d'environnement cohérente et une répétabilité.

google-drive-ocamlfuse is a FUSE-based filesystem that mounts a Google Drive account as a local directory, enabling standard file operations on cloud files. It bridges POSIX filesystem calls to the Google Drive API, allowing users to read, write, and manage files through their operating system's native file manager or command line. The project distinguishes itself through support for multiple simultaneous Google Drive accounts, each mounted as an independent local directory with separate authentication and cache state. It handles Google Docs, Sheets, and Slides by exporting them as read-only

CubeFS est un système de stockage cloud distribué conçu pour gérer le stockage de fichiers et d'objets à travers des centres de données et des clouds hybrides. Il fonctionne comme un système de fichiers distribué multi-tenant et un magasin d'objets capable de gérer des données à l'échelle de l'exaoctet, utilisant une architecture distribuée pour stocker du contenu non structuré. Le système se distingue par une couche d'interface multi-protocole qui permet un accès simultané aux données via les interfaces S3, POSIX et HDFS. Il emploie une architecture découplée calcul-stockage pour faire évoluer le traitement et la persistance indépendamment, et implémente des politiques d'isolation fine pour séparer les ressources et les données entre les différents tenants. La fiabilité est gérée par des stratégies de redondance configurables, incluant la mise en miroir multi-répliques et le codage à effacement (erasure coding). La plateforme inclut un système de mise en cache multi-niveaux pour accélérer l'accès aux données et s'intègre à Kubernetes via un pilote Container Storage Interface pour automatiser le provisionnement de volumes persistants.

Mountpoint for Amazon S3 is a FUSE-based filesystem client that mounts S3 buckets as local directories, enabling standard file operations on objects without custom code. It enforces S3 bucket permissions through AWS Identity and Access Management policies on every operation, and implements lazy object materialization to fetch content on-demand rather than downloading entire objects at mount time. The filesystem maps S3's flat key namespace into a hierarchical directory structure using forward slashes as path separators, and supports write-back object assembly that accumulates local writes into

Cyberduck is a multi-protocol file transfer client and cross-platform file manager for Mac and Windows. It functions as a cloud storage manager and remote storage mounter, allowing users to upload, organize, and synchronize data between local drives and remote endpoints. The application provides a unified interface for managing files across FTP, SFTP, WebDAV, S3, and other cloud storage protocols, including Amazon S3, Backblaze B2, Microsoft Azure, and OneDrive. It distinguishes itself with a client-side encryption vault that encrypts files and folders locally before they are transmitted to r

img2dataset is a high-performance image dataset pipeline and preprocessing tool designed to download and process millions of images from URLs for machine learning training. It functions as a distributed image downloader and cloud storage data exporter, moving large visual datasets from web sources directly into structured formats. The system prioritizes high-throughput data acquisition by distributing workloads across multiple CPU cores and machines. It integrates directly with remote cloud storage buckets and employs a manifest-based tracking system to resume interrupted downloads without re

JimsGarage is a collection of shell scripts and automation tools designed to help individuals deploy and manage a wide range of self-hosted services on their own hardware. It provides a structured approach to setting up containerized applications, from media servers and document management systems to VPNs and monitoring stacks, all through automated Docker-based configurations. The project distinguishes itself by offering a comprehensive library of deployment recipes that cover the full lifecycle of a home server environment. This includes not just the services themselves, but also the suppor

Ce projet est un bot miroir cloud Telegram conçu pour télécharger des fichiers depuis diverses sources internet et les refléter sur Telegram ou sur un stockage cloud. Il fonctionne comme un service spécialisé pour le téléchargement de médias à distance, le leeching de torrents et Usenet, et le mirroring de contenu automatisé. Le bot se distingue par une intégration profonde avec l'écosystème Rclone pour gérer, cloner et migrer des fichiers à travers plusieurs fournisseurs cloud. Il inclut un automatiseur de contenu RSS pour déclencher des téléchargements basés sur des filtres définis par l'utilisateur et utilise la rotation de comptes de service pour contourner les quotas d'API lors de la gestion des ressources cloud. Le système couvre un large éventail de capacités, incluant le traitement multimédia pour l'extraction d'archives et la conversion de format, ainsi que la capture de médias web depuis des plateformes prises en charge. Il propose également la découverte de torrents via des API externes, la recherche récursive de contenu cloud et une file d'attente de tâches pour gérer les travaux de transfert à haut volume. Le contrôle administratif est assuré par un contrôle d'accès basé sur le chat et une base de données pour persister les configurations utilisateur et l'historique des tâches.

CML est un outil d'automatisation de pipeline pour l'entraînement et l'évaluation de modèles d'apprentissage automatique, fonctionnant comme un système CI/CD pour l'apprentissage automatique. Il sert d'orchestrateur de calcul cloud et de gestionnaire de flux de travail basé sur Git qui automatise les cycles d'entraînement de modèles via la gestion de branches, les commits automatisés et le reporting intégré. Le projet se distingue par le provisionnement d'instances cloud éphémères ou de nœuds Kubernetes pour fournir du matériel spécialisé pour les tâches gourmandes en calcul. Il gère également des exécuteurs de calcul distants, permettant la connexion de clusters GPU auto-hébergés ou de machines sur site pour exécuter des flux de travail d'apprentissage automatique conteneurisés. Le système couvre un large éventail de capacités, incluant le suivi des expériences ML, où les métriques de performance et les visualisations sont publiées directement dans les pull requests de contrôle de version. Il gère l'automatisation du pipeline ML depuis l'importation initiale des données et le versionnage jusqu'à la génération de rapports de flux de travail formatés et de liens de visualisation externes. L'outil fournit une utilité supplémentaire pour la gestion de l'infrastructure via le débogage distant basé sur SSH et la capacité de reprendre les tâches interrompues.