Projects sharing features with Dragonfly

This project is a full-featured BitTorrent client library and protocol implementation. It provides the core components necessary for peer-to-peer file sharing, including a networking stack that supports the BitTorrent specification, tracker communication, and swarm management. A primary differentiator is the inclusion of a virtual filesystem that allows torrents to be mounted as read-only drives. This system enables on-demand content streaming, where specific data pieces are fetched from the network only when a read request occurs, rather than requiring the entire archive to be downloaded fir

Adaptive Accrual Failure Detector

A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, provides a health endpoint that exposes their status, and health metrics.

This project is a curated knowledge repository that aggregates high-quality resources, technical documentation, and expert insights focused on distributed systems engineering. It serves as a community-driven learning resource designed to help developers navigate the complexities of building and maintaining large-scale software applications. The repository distinguishes itself through a hierarchical taxonomy that organizes vast amounts of technical information into a structured, searchable format. By utilizing markdown-based content curation and static indexing, the collection remains version-

This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It provides a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams. The library distinguishes itself through a comprehensive toolkit for shell-like operations, including a text processing engine for regular expression filtering and frequency analysis, a filesystem utility toolkit for recursive search and path manipulation, and an integrated HTTP client wrapper for building data pipelines that fetch web content. The capability s

Simplified distributed locking implementation using Redis

Build Your Own Kafka

Consistent is a Go library that implements consistent hashing with bounded loads to distribute data keys across nodes in a distributed system. It provides a mechanism for mapping keys to cluster members that minimizes data movement during membership changes while preventing performance hotspots. The library distinguishes itself by enforcing strict capacity limits on individual nodes, ensuring that no single member becomes overwhelmed by excessive key assignments. It supports virtual node mapping to distribute physical capacity across the hash ring, allowing for granular control over load bala

Distributed batch data processing framework

🌧 BitTorrent client and library in Go

Run a command when files change

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Kitex is a high-performance remote procedure call framework for building distributed microservices in Go. It provides a communication system compatible with gRPC and Protobuf standards and includes a suite of tools for microservices governance. The framework features proxyless service mesh integration, which allows it to manage network traffic and governance directly without adding sidecar proxy latency. It optimizes data throughput through a high-performance communication library that minimizes memory copies. The system covers a broad surface of distributed capabilities, including service d

High-performance Distributed Command Pattern (DCP) for Go

The best HTTP Static File Server, write with golang+vue

Containerd is a daemon-based container runtime that manages the complete lifecycle of containers on a host system. It functions as a core orchestration backend, handling image distribution, storage, and process execution while adhering to industry-standard specifications for container execution and configuration. The project is distinguished by its modular, plugin-based architecture, which allows for the extension of storage, runtime, and networking capabilities without requiring a full daemon recompile. It utilizes a shim-based execution model to delegate low-level operations, ensuring isola

etcd is a distributed key-value store and configuration store designed to maintain a consistent set of data across a cluster of nodes. It functions as a reliable registry for storing and synchronizing critical settings and metadata used by distributed applications. The system implements the Raft consensus algorithm to ensure data consistency and leader election across servers. To protect data transfers and verify node identities, it utilizes a network security layer based on mutual TLS and client certificates. Its capabilities cover distributed configuration management, cluster state synchro

Compute cluster (HPC) job submission library for Go (#golang) based on the open DRMAA standard.

go-jump: Jump consistent hashing