Awesome GitHub RepositoriesCloud-Native Storage Layers

MinIO is a software-defined, cloud-native object storage server designed to manage large volumes of unstructured data. It functions as a distributed storage cluster that aggregates multiple independent nodes into a unified, scalable pool, providing a high-performance infrastructure compatible with standard cloud storage protocols and application programming interfaces. The system utilizes a shared-nothing architecture that eliminates central metadata servers, relying instead on a decentralized hash table to map objects across the cluster. Data availability and resilience are maintained throug

go-ipfs is an implementation of an IPFS node, providing a distributed filesystem and a content-addressable storage system. It enables the storage and retrieval of data based on unique cryptographic hashes rather than fixed network locations, allowing files to be shared across a peer-to-peer network without a central authority. The system utilizes a distributed hash table and a peer-to-peer gossip protocol to route requests and propagate network state and metadata. It organizes data using a Merkle DAG structure to support efficient deduplication and versioning of content. Capabilities include

CVAT is an open-source, web-based platform designed for annotating images, videos, and 3D point clouds to create high-quality training datasets for machine learning. It functions as a containerized server that orchestrates the entire lifecycle of computer vision data, from initial task creation and manual labeling to quality assurance and final dataset export. The platform distinguishes itself through deep integration with machine learning models, allowing users to deploy custom AI models as serverless functions for automated object detection, tracking, and skeleton annotation. It supports co

JuiceFS is a distributed file system designed to mount object storage as a local, POSIX-compliant drive. It functions as a cloud-native persistent storage layer that decouples file metadata from raw data, storing metadata in a transactional database while keeping data blocks in object storage. This architecture enables multiple hosts across different regions to access the same storage simultaneously while maintaining strong consistency. The system distinguishes itself by performing data processing, including compression and encryption, directly on the client side before transmission. By split

SkyPilot is a multi-cloud AI orchestrator and distributed task scheduler designed to launch and manage AI workloads across various cloud providers, Kubernetes, and Slurm clusters. It functions as an infrastructure-as-code framework that uses declarative files to define resource requirements and setup commands for consistent execution across different environments. The project differentiates itself through automated cost optimization, selecting the most affordable GPU or TPU hardware and managing spot instances to reduce expenses. It also provides a remote development environment that bridges

KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models on personal hardware. It functions as a multimodal AI server and API gateway, providing OpenAI-compatible endpoints that allow third-party clients to interact with locally hosted models. The project distinguishes itself as an AI storytelling backend, featuring dedicated tools for long-form narrative management through persistent memory, world lore tracking, and character state management. It further extends its capabilities as a multimodal server capable of processing text, im

all-in-one is a containerized deployment system designed to install and manage a complete suite of productivity and collaboration services. It functions as a cloud suite deployer that orchestrates the installation of a self-hosted content platform, incorporating necessary dependencies via Docker or Kubernetes. The project distinguishes itself by providing a web-based dashboard for orchestrating, updating, and monitoring the lifecycle of service containers. It also serves as a local AI inference server, enabling the execution of generative text models, image diffusion, and speech processing on

h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

This is an open-source educational website that translates and localizes MIT's Missing Semester course, teaching practical computing skills for computer science students. The curriculum covers developer tooling, shell scripting, version control, security fundamentals, and open-source collaboration, with a focus on core computing skills including data processing pipelines, workflow automation, secure remote access, shell productivity, Vim editing, and Git version control. The project distinguishes itself by teaching command-line mastery, shell scripting, and automation to boost daily developer

ZeroByte is a backup management platform built around the Restic backup engine, providing encrypted, deduplicated, and compressed snapshots across multiple storage backends. It offers a web interface for scheduling, monitoring, and managing backup operations, with support for cron-based job scheduling and configurable retention policies that automatically prune older snapshots. The platform distinguishes itself through comprehensive multi-protocol volume mounting, allowing backup ingestion from NFS, SMB, WebDAV, SFTP, and rclone-backed sources alongside local directories. It includes a snapsh

SSHFS-Win is a Windows implementation of SSHFS that mounts remote directories over SSH as local Windows drives, enabling seamless file access as if they were local network drives. It provides both command-line and graphical interfaces for creating, managing, and disconnecting SSHFS mounts, supporting password or SSH key authentication with optional credential storage in the Windows Credential Manager. The project extends beyond basic SSH mounting to support a wide range of remote file access scenarios, including mounting cloud storage services like Azure Blob or Amazon S3, distributed POSIX f

该项目是 AWS Lambda 的性能优化器和资源基准测试工具。它通过测试各种内存配置来分析执行速度与成本之间的权衡，以确定最具成本效益的设置并最小化运营支出。 该工具利用 AWS Step Functions 编排器来自动化执行和收集跨不同功率级别的多个函数测试运行的数据。它通过注入自定义静态或远程数据并使用加权有效负载分布来模拟生产工作负载，以模仿真实世界的流量模式。 该套件涵盖了几个功能领域，包括迭代内存采样和基于指标的成本建模，以可视化性能权衡。它为临时函数版本和别名提供自动化资源清理、为受限内部资源提供私有网络配置，以及提供远程有效负载加载以绕过标准调用大小限制。 部署通过基础设施即代码（IaC）结构进行处理，以确保环境设置的一致性和可重复性。

google-drive-ocamlfuse is a FUSE-based filesystem that mounts a Google Drive account as a local directory, enabling standard file operations on cloud files. It bridges POSIX filesystem calls to the Google Drive API, allowing users to read, write, and manage files through their operating system's native file manager or command line. The project distinguishes itself through support for multiple simultaneous Google Drive accounts, each mounted as an independent local directory with separate authentication and cache state. It handles Google Docs, Sheets, and Slides by exporting them as read-only

CubeFS 是一个分布式云存储系统，旨在管理跨数据中心和混合云的文件和对象存储。它作为一个多租户分布式文件系统和对象存储，能够处理艾字节（exabyte）规模的数据，并利用分布式架构存储非结构化内容。 该系统以其多协议接口层为特色，允许通过 S3、POSIX 和 HDFS 接口同时访问数据。它采用存算分离架构以独立扩展处理和持久化能力，并实施细粒度的隔离策略以分离不同租户间的资源和数据。 可靠性通过可配置的冗余策略进行管理，包括多副本镜像和纠删码。该平台包含一个多级缓存系统以加速数据访问，并与 Kubernetes 通过容器存储接口（CSI）驱动程序集成，以实现持久卷的自动化配置。

Mountpoint for Amazon S3 is a FUSE-based filesystem client that mounts S3 buckets as local directories, enabling standard file operations on objects without custom code. It enforces S3 bucket permissions through AWS Identity and Access Management policies on every operation, and implements lazy object materialization to fetch content on-demand rather than downloading entire objects at mount time. The filesystem maps S3's flat key namespace into a hierarchical directory structure using forward slashes as path separators, and supports write-back object assembly that accumulates local writes into

Cyberduck 是一个多协议文件传输客户端和适用于 Mac 和 Windows 的跨平台文件管理器。它作为一个云存储管理器和远程存储挂载器，允许用户在本地驱动器和远程端点之间上传、组织和同步数据。 该应用提供了一个统一的界面，用于管理 FTP、SFTP、WebDAV、S3 以及其他云存储协议（包括 Amazon S3、Backblaze B2、Microsoft Azure 和 OneDrive）上的文件。它的特点是拥有一个客户端加密库，可在文件传输到远程服务器之前在本地对文件和文件夹进行加密。 该系统支持远程文件管理和跨协议文件传输，包括将远程服务器或云目录直接挂载到本地操作系统文件资源管理器的功能。其他功能包括用于执行文件传输任务的命令行接口。

img2dataset 是一个高性能图像数据集流水线和预处理工具，旨在为机器学习训练从 URL 下载并处理数百万张图像。它作为一个分布式图像下载器和云存储数据导出器，将大型视觉数据集从 Web 源直接移动到结构化格式中。 该系统通过在多个 CPU 核心和机器之间分配工作负载，优先考虑高吞吐量的数据获取。它直接与远程云存储桶集成，并采用基于清单的追踪系统，无需重新处理现有数据即可恢复中断的下载。 该工具为机器学习数据集准备提供了完整的预处理套件，包括图像缩放、裁剪以及基于尺寸或长宽比的属性过滤。它还通过哈希比较验证图像完整性，并在抓取工作流期间确保符合机器人协议。 该项目使用 Python 实现。

JimsGarage is a collection of shell scripts and automation tools designed to help individuals deploy and manage a wide range of self-hosted services on their own hardware. It provides a structured approach to setting up containerized applications, from media servers and document management systems to VPNs and monitoring stacks, all through automated Docker-based configurations. The project distinguishes itself by offering a comprehensive library of deployment recipes that cover the full lifecycle of a home server environment. This includes not just the services themselves, but also the suppor

本项目是一个 Telegram 云镜像机器人，旨在从各种互联网源下载文件并将其镜像到 Telegram 或云存储。它作为一个专门用于远程媒体下载、种子和 Usenet 离线下载以及自动内容镜像的服务。 该机器人通过与 Rclone 生态系统的深度集成来管理、克隆和迁移跨多个云服务商的文件，从而脱颖而出。它包括一个 RSS 内容自动化程序，可根据用户定义的过滤器触发下载，并利用服务账户轮换在管理云资源时绕过 API 配额。 该系统涵盖了广泛的功能，包括用于归档提取和格式转换的媒体处理，以及从支持平台进行 Web 媒体捕获。它还具有通过外部 API 进行种子发现、递归云内容搜索以及用于管理大容量传输任务的任务队列。 管理控制通过基于聊天的访问控制以及用于持久化用户配置和任务历史的数据库提供。

CML 是一个用于训练和评估机器学习模型的管道自动化工具，作为机器学习的 CI/CD 系统运行。它作为一个云计算编排器和基于 Git 的工作流管理器，通过分支管理、自动提交和集成报告来自动化模型训练周期。 该项目通过配置临时云实例或 Kubernetes 节点来提供计算密集型任务所需的专用硬件，从而脱颖而出。它还管理远程计算运行器，允许连接自托管 GPU 集群或本地机器来执行容器化机器学习工作流。 该系统涵盖了广泛的功能，包括 ML 实验跟踪（性能指标和可视化直接发布到版本控制 Pull Request 中）。它处理从初始数据导入和版本控制到生成格式化工作流报告和外部可视化链接的 ML 管道自动化。 该工具通过基于 SSH 的远程调试和恢复中断作业的能力，为基础设施管理提供了额外的实用性。