dragonflyoss/dragonfly-archivedArchived
Dragonfly Archived
Dragonfly is a peer-to-peer file distribution system designed to accelerate large-scale file and container image delivery across clusters. It splits files into small pieces that peers exchange independently, enabling parallel downloads from multiple sources while a central scheduler assigns parent peers based on real-time load metrics. The system integrates with existing tools through an HTTP proxy interface and uses gRPC for internal communication between peers and the scheduler.
The system distinguishes itself through automatic integrity verification at the piece level, ensuring every downloaded file matches the original without manual checks. It provides fault isolation at multiple levels—service, peer, or task—so a single failure cannot cascade across the network. Per-host bandwidth throttling caps download speed at the machine level, limiting total bandwidth used by all concurrent tasks on that host, while passive CDN caching stores downloaded files on content delivery nodes to serve subsequent requests without contacting the original source.
Dragonfly includes a web-based cluster management console for configuring, monitoring, and managing peer-to-peer distribution clusters and their activity. It supports container image acceleration by distributing layers through the P2P network instead of pulling directly from a registry, and authenticates file source requests through standard HTTP header fields. The system also automatically detects and excludes malfunctioning peers or managers from the distribution network to maintain download reliability.
Features
- Peer-to-Peer File Distribution - Uses peer-to-peer networking to transfer files, leveraging each peer's bandwidth to improve download efficiency and reduce cross-datacenter traffic.
- Piece-Level File Transfer Protocols - Files are split into small pieces that peers exchange independently, enabling parallel downloads from multiple sources.
- Service-Level Isolations - Errors are contained within individual services, peers, or tasks so a single failure cannot cascade across the network.
- Peer-to-Peer Download Schedulers - Central scheduler assigns parent peers to each download based on real-time load metrics across the cluster.
- P2P Image Accelerators - Speeding up container image pulls by distributing layers through a P2P network instead of pulling directly from a registry.
- Peer-to-Peer Image Distributions - Accelerates large-scale file and image distribution by having peers share downloaded pieces with each other, reducing source load.
- Container Registry Proxies - Acts as an HTTP/HTTPS proxy that accelerates container image pulls from registries without modifying existing tools.
- Cluster Bandwidth Controllers - Controlling and balancing download bandwidth across a cluster to prevent network congestion and ensure fair resource usage.
- Peer-to-Peer Download Accelerators - Speeds up file downloads by having peers share pieces of the file with each other, reducing load on the original source.
- Per-Host Bandwidth Limiters - Download speed is capped at the machine level, limiting total bandwidth used by all concurrent tasks on that host.
- HTTP Proxies - Existing tools connect through an HTTP proxy interface, requiring no modifications to use the P2P distribution network.
- File Integrity Verifications - Guarantees that every downloaded file matches the original without requiring the user to supply a checksum.
- Download Integrity Verification - Verifies file consistency automatically during P2P transfers so users receive correct content without manual checks.
- Integrity Verification - Each file piece carries a hash that the receiving peer checks immediately, guaranteeing end-to-end correctness.
- Download Failure Isolations - Contains errors at the service, peer, or task level to prevent a single failure from disrupting other transfers.
- Proxy-Based Tool Integrations - Works with container runtimes, registries, and download tools through HTTP/HTTPS proxy or gRPC without modifying them.
- Parent Peer Schedulers - Selects optimal parent peers for each download based on real-time load, balancing bandwidth across the cluster.
- CDN Offloading - Reducing load on origin servers by caching downloaded files on passive CDN nodes and serving them to peers.
- Automatic Peer Exclusions - Detects and excludes malfunctioning peers or managers from the distribution network to maintain download reliability.
- Per-Host Bandwidth Caps - Restricts the total download bandwidth used by all tasks on a single machine, not just per individual download.
- gRPC Interfaces - Internal communication between peers and the scheduler uses gRPC for efficient, typed message passing.
- File Transfer Fault Isolators - Ensuring reliable file distribution by isolating failures at the service, peer, or task level and automatically excluding faulty nodes.
- Cluster Management Interfaces - Provides a web interface to configure, monitor, and manage peer-to-peer distribution clusters and their activity.
- P2P Cluster Consoles - Managing and monitoring a peer-to-peer distribution cluster through a visual console for configuration and activity tracking.
- Passive Caching Nodes - Downloaded files are stored on a content delivery node that serves subsequent requests without contacting the original source.
