Awesome GitHub RepositoriesHigh-Performance Data Transfer

bypy is a Python client for interacting with Baidu Yun cloud storage via its official API. It functions as both a library for programmatic integration and a command-line manager for organizing and listing files within a cloud account. The project features a multi-process transfer tool that utilizes parallel processing to accelerate high-volume uploads and downloads. It provides bidirectional directory synchronization to align content between local file systems and remote cloud storage by identifying missing or outdated files. The tool covers recursive file transfers, automated cloud data bac

pgBackRest est un outil de sauvegarde et de restauration distribué pour PostgreSQL. Il fonctionne comme un moteur de sauvegarde incrémentale et un gestionnaire de récupération à un point dans le temps, permettant la création de sauvegardes complètes, différentielles et incrémentales à travers des dépôts locaux et distants. Le système se distingue par ses sauvegardes incrémentales au niveau bloc, qui réduisent le stockage et le temps de transfert en enregistrant uniquement les données modifiées. Il s'intègre avec le stockage d'objets compatible S3 via HTTP et utilise le traitement parallèle pour maximiser le débit pendant les opérations de sauvegarde et de restauration. L'outil couvre des capacités complètes de reprise après sinistre, incluant l'archivage des journaux de transactions (WAL) pour la restauration à un point dans le temps et la vérification d'intégrité basée sur des sommes de contrôle pour détecter la corruption des données. Il fournit également une automatisation du cycle de vie des sauvegardes via une rotation pilotée par politique et sécurise les données via le chiffrement du dépôt et des protocoles de transport sécurisés.

s5cmd is a command line interface for high-performance data transfers and management tasks across S3 compatible storage services. It functions as a parallel data transfer tool and bucket synchronization utility, designed to accelerate the uploading and downloading of large volumes of files using concurrent workers. The tool acts as a batch command processor capable of executing thousands of object management operations in parallel from command files or piped streams. It also serves as an S3 Select query client, allowing the execution of SQL expressions against stored JSON records to retrieve

Teldrive is a web-based cloud storage interface that uses Telegram as a backend for storing and organizing files. It functions as an encrypted cloud drive and a remote storage bridge, allowing users to manage files through a dedicated file manager. The system optimizes data throughput by using a proxy to aggregate multiple authentication tokens, which bypasses single-account rate limits to increase upload and download speeds. It ensures data privacy through per-file salt encryption for both file data and metadata. The project covers remote file management, including the ability to mount stor

Connector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally. The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar a