Awesome GitHub RepositoriesStorage Throughput Optimizers

RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware. The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugin

ZFS is an enterprise-grade file system and logical volume manager that integrates storage pooling with advanced data protection. It functions as a storage engine that aggregates multiple physical devices into a unified resource pool, allowing for the dynamic allocation of capacity across individual file systems. The system utilizes a transactional, copy-on-write architecture that ensures file system consistency through intent logging and atomic operations. It maintains data integrity by organizing blocks into a hierarchical tree structure, where cryptographic checksums are used to detect and

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Este proyecto es una colección de herramientas de utilidad para ampliar la compatibilidad de hardware en sistemas Synology NAS. Proporciona una herramienta de compatibilidad de unidades para añadir discos duros, SSDs y unidades NVMe no compatibles a la base de datos interna del sistema, junto con un supresor de alertas de hardware para ocultar notificaciones de memoria no certificada y discos de terceros. El kit de herramientas incluye un desbloqueador de almacenamiento específicamente para habilitar pools y volúmenes de almacenamiento M.2 PCIe en tarjetas de hardware no compatibles. También cuenta con un optimizador de RAM para ajustar los límites máximos de memoria y optimizar la RAM reservada para cachés de disco. Las capacidades adicionales cubren la optimización del rendimiento del almacenamiento, como la configuración de prioridades de lectura de disco y la habilitación del soporte TRIM para aumentar el rendimiento de datos. El proyecto también proporciona actualizaciones para herramientas de monitoreo de salud del hardware para asegurar que los modelos de unidades de terceros más nuevos informen el estado correctamente.

Ghost-Downloader-3 is a multithreaded download manager and HTTP file downloader designed to accelerate file transfers. It functions as a browser-integrated download client, bridging web extensions to a local application to automate task capture and transfer. The system utilizes a multithreaded transfer engine that employs parallel chunking to maximize bandwidth. It distinguishes itself through a concurrent transfer orchestrator that manages simultaneous download queues, provides dynamic bandwidth throttling, and supports proxy server routing to manage network paths. The application includes

InternetArchitect es una colección educativa de documentos y código fuente diseñada como un curso de arquitectura de alta concurrencia. Sirve como una guía de implementación de sistemas distribuidos, proporcionando patrones técnicos y ejemplos prácticos para diseñar arquitecturas de internet escalables que mantengan la estabilidad bajo cargas de tráfico pesadas. El proyecto se centra en la optimización de bases de datos de alto rendimiento y patrones de diseño de microservicios. Cubre estrategias para reducir la latencia y aumentar el rendimiento a través de sharding de bases de datos y capas de proxy, así como la coordinación del estado global a través de clústeres distribuidos. El alcance arquitectónico incluye estrategias de almacenamiento en caché multinivel para acelerar la recuperación de datos y la implementación de frameworks de descubrimiento de servicios para gestionar la comunicación entre microservicios desacoplados. También aborda la coordinación de estado distribuido y el uso de mallas de balanceo de carga para distribuir el tráfico de red a través de servidores backend.

Connector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally. The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar a

This project is an OCI-compatible container runtime that executes workloads within lightweight virtual machines. By leveraging hardware-based virtualization, it provides strong security isolation between containerized processes and the host operating system, serving as a drop-in replacement for traditional container execution environments. The runtime distinguishes itself through a hypervisor-agnostic architecture that abstracts underlying virtualization operations, allowing for consistent container lifecycle management across different backends. It integrates directly with standard container