14 个仓库
Techniques for decoupling high-frequency write operations from primary storage using queues to prevent saturation.
Distinct from Write Skew Mitigations: None of the candidates describe the architectural pattern of using queues to decouple writes from databases.
Explore 14 awesome GitHub repositories matching data & databases · Asynchronous Write Buffering. Refine with filters or upvote what's useful.
This project is a comprehensive Java backend engineering guide and technical reference focused on high-concurrency design, distributed systems, and microservices architecture. It provides detailed strategies for decomposing monolithic applications, managing service discovery, and implementing the architectural patterns required for scalable backend environments. The repository distinguishes itself through an extensive collection of big data algorithmic references and database scaling strategies. It covers memory-efficient techniques for analyzing massive datasets, such as Top-K element extrac
Provides patterns for buffering high-frequency write requests in queues to prevent database saturation.
Mars is a cross-platform networking SDK and high-performance logging framework. It provides a network reliability layer for managing short and long-lived connections across different operating systems, alongside a diagnostic tool for recording system events to disk with minimal application overhead. The project features a persistent connection manager that maintains bidirectional server links using heartbeats and sequence tracking. It includes a traffic control engine to prevent server request avalanches by limiting the frequency and volume of outgoing requests. To improve connectivity, it ut
Implements a high-speed writing mechanism that separates log generation from disk I/O to minimize application overhead.
Betaflight 是一款开源飞控固件,旨在通过 PID 回路和电机控制来稳定飞行器并管理传感器数据。它作为一个多旋翼飞行栈,集成了底层驱动程序和控制算法,用于管理电子调速器(ESC)、无线电接收机和遥测硬件。 该系统专注于实时飞行稳定和遥测路由。它包括一个用于计算电机输出以保持飞行器稳定性的 PID 稳定系统,以及一个用于将实时信息路由到地面站和屏幕显示(OSD)的飞行遥测管理器。 该固件涵盖了广泛的操作功能,包括飞行性能调优、硬件外设配置和串行通信路由。它提供了监控和可观测性工具,如高频飞行数据记录和抬头显示(HUD)生成,以及用于在没有物理硬件的情况下验证固件行为的物理仿真功能。
Implements high-speed memory buffers to decouple high-frequency sensor logging from slower physical storage writes.
RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen
Decouples streaming processes from downstream write speeds using buffers to prevent pipeline blocks.
This is a high-performance C++ HTTP web server designed for delivering static content. It operates as an asynchronous event loop server that combines IO multiplexing with worker thread pools to handle concurrent requests without blocking. The server specializes in TCP connection management, featuring keep-alive sessions, request pipelining, and timer-based timeout tracking for idle sockets. To ensure high throughput, it utilizes double-buffered asynchronous logging and background threads to prevent disk I/O latency from impacting network traffic. The system covers broad capability areas incl
Uses a double-buffered asynchronous logging system to record server events without interrupting the primary execution flow.
This project is a machine learning experiment tracker and event file generator that enables the recording of scalars, images, and histograms to monitor model performance. It functions as an integration bridge that allows training metrics from PyTorch to be logged into files compatible with the TensorBoard dashboard. The system includes a remote log synchronizer designed to stream experiment data to cloud services. This allows for the remote management and analysis of training results and the comparison of datasets across different training runs. The utility covers a broad range of monitoring
Uses high-speed memory buffers to decouple training log generation from physical disk I/O.
SSHFS is a network filesystem client that maps remote server directories to local mount points. It functions as a userspace implementation of a filesystem, allowing users to access and manage remote files as if they were on a local disk using the SFTP protocol over SSH. The tool distinguishes itself by offering multiple transport options, including encrypted SSH tunnels for security and direct TCP socket connections to bypass encryption for higher throughput. It also supports connectivity via virtual sockets to access directories hosted inside virtual machines. The system includes capabiliti
Implements asynchronous write buffering to decouple local file writes from remote server synchronization for improved performance.
Consola is a diagnostic logging utility and log output manager that provides a unified interface for Node.js and web browser environments. It functions as a scoped logging framework and a tool for capturing user text, confirmations, and selections through interactive console prompts. The project distinguishes itself through a system for creating specialized logger instances with inherited defaults and unique tags for contextual tracking. It also features a pluggable reporter interface that allows for the redirection of standard output to custom logging destinations and external reporters. Th
Buffers messages temporarily during system pauses and emits them upon resumption.
Pulse is an Apple platform network logger used for recording and inspecting network traffic and system logs directly within applications. It functions as an on-device network inspector and traffic export tool, allowing for the capture of outgoing requests and the transmission of logs to external systems. The project includes a remote log streamer for the real-time transmission of device logs to an external viewer for live monitoring. It further provides an integrated interface for viewing recorded network requests and system logs on the device without the requirement of an external proxy. Th
Aggregates stored event data into a transferable format for extraction to external debugging tools.
xctool 是一个专为 iOS 和 macOS 持续集成设计的 xcodebuild 命令行包装器。它作为一个并行测试运行器、构建日志处理器和报告生成器,用于自动化 Apple 平台的构建和测试流水线。 该工具将测试包分发到多个 CPU 核心以减少执行时间,并提供通过过滤特定方案、类或方法来运行测试子集的能力。它通过允许命令行参数从 JSON 配置文件中持久化和加载,简化了构建管理。 它将冗长的构建日志转换为结构化的 JSON 事件流,这些流可以转换为 JUnit XML 以与外部报告工具兼容。为了提高可观测性,它使用了一个日志抑制缓冲区,在检测到错误之前隐藏输出,提供人类可读且带颜色的控制台文本。
Holds build output in a temporary buffer and only prints to the console when an error is detected.
Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis
Decouples high-frequency write operations from primary storage using buffers to improve performance for write-heavy workloads.
Logan is a cross-platform mobile logging framework that collects, stores, and uploads client-side logs from iOS, Android, Web, and Flutter environments for centralized debugging and analysis. It provides a complete pipeline from client-side log buffering and file-based local storage through to server-side ingestion and a visual browser for inspecting parsed logs. The system uses a structured binary protocol to encode log entries with content, type, timestamp, and thread metadata, enabling consistent parsing across platforms. A log receiving server handles uploaded files, while a web-based int
Logs are written to local storage in a buffered manner before being uploaded to a remote server for analysis.
Fleet is an open-source device management platform that provides centralized control over computing devices running macOS, Linux, Windows, Chromebooks, iOS, and Android. It enables organizations to enroll devices, collect real-time telemetry, enforce security compliance policies, and manage software remotely from a single system. The platform can be deployed as a single binary, run locally for testing, or scaled horizontally across cloud infrastructure on AWS, Kubernetes, GCP, or Render, with support for high availability through database replication and load balancing. The platform distingui
Continues executing configuration and buffering result logs on osquery agents when network connectivity is lost.
EasyLogger 是一个轻量级的 C/C++ 日志库和异步框架,专为嵌入式系统、物联网设备和可穿戴设备设计。它作为资源高效的记录器和闪存管理器,为低级调试和系统事件记录提供了工具,且对 ROM 和 RAM 的要求极低。 该项目的特色在于异步日志模型,将消息缓冲到独立线程,防止日志操作阻塞实时应用执行。它包含一个专门的闪存管理器,利用循环缓冲将日志持久化到非易失性存储中,确保关键数据在设备重启后得以保留。 该库涵盖了广泛的可观测性功能,包括用于状态验证的系统断言、编译时和运行时的严重性过滤,以及原始二进制十六进制转储的生成。它支持通过时间戳和线程标识符进行元数据增强,并提供了一个可移植接口,用于将日志路由到串口或文件等自定义目的地。 该系统通过写入同步确保数据完整性,并提供了手动缓冲区刷新和存储日志检索的机制。
Decouples log generation from physical writing using high-speed RAM buffers and a background thread.