65 dépôts
Techniques for staging data in volatile memory to optimize disk I/O performance.
Distinct from Sorted Memory Buffers: General memory buffering for file I/O, distinct from LSM-tree sorted buffers or terminal grids.
Explore 65 awesome GitHub repositories matching software engineering & architecture · Memory Buffering. Refine with filters or upvote what's useful.
Excelize is a Go library designed for reading, writing, and modifying Microsoft Excel files in XML-based formats. It functions as a spreadsheet file parser and generator that enables the programmatic extraction and modification of data. The library includes a streaming spreadsheet processor to handle massive datasets incrementally, preventing system memory exhaustion during large-scale read and write operations. It also provides a chart generator to convert worksheet values or external data sources into visual representations within the spreadsheet. Beyond core file processing, the project c
Implements memory buffers to stage spreadsheet data and optimize disk access patterns.
MMKV is a high-performance, cross-platform key-value storage framework designed for mobile platforms and POSIX environments, including Android, iOS, macOS, and Windows. It provides a persistence layer that utilizes memory-mapped files and binary serialization to achieve low-latency data access. The project distinguishes itself through native support for multi-process synchronization, allowing concurrent read and write operations across different application processes. It also implements security via AES encryption for data at rest, featuring symmetric encryption and key rotation to protect st
Optimizes performance by passing large values using native memory buffers to avoid expensive data copying.
KCP is a low-latency transport protocol and reliability layer designed to provide TCP-like guaranteed delivery and ordering over UDP. It implements an automatic repeat request protocol to minimize network jitter and delays, prioritizing timeliness over bandwidth efficiency. The project distinguishes itself through a pluggable congestion controller and flow control logic, allowing the transport layer to be tuned for specific network environments. It includes a NAT keep-alive system that uses periodic heartbeat packets to maintain active communication paths and prevent address translation timeo
Manages transmission throughput and memory usage via configurable window-based buffering for send and receive queues.
Perfect Green Screen Keys
Decodes video frames into a temporary buffer for individual processing before re-encoding the output stream.
Assimp is a cross-platform 3D asset pipeline and import library that loads numerous industry-standard 3D file formats into a single unified internal data structure. It functions as a framework for converting 3D models between different file formats across multiple operating systems and architectures. The project provides a 3D mesh processing tool for normalizing and optimizing geometry through triangulation, vertex removal, and normal generation. It also includes a 3D asset export utility to write internal scene data back into various external file formats. The system covers broad capability
Allows 3D model data to be imported directly from memory buffers.
Twemproxy is a lightweight proxy that routes and distributes requests across multiple Redis and Memcached backend servers. It functions as a protocol translation gateway and distributed cache shard manager, partitioning data across clusters to balance load and storage capacity. The system acts as a high-availability cache orchestrator, employing health monitoring and automatic server ejection to maintain continuous access to cached data. It integrates with sentinels for dynamic master and replica discovery and utilizes consistent hashing and tag-based key grouping to manage data distribution
Adjusts memory buffer sizes for zero-copy data transfers to balance concurrent connections against data throughput.
Multer is a multipart form parser and file upload manager designed as middleware for Node.js and Express.js applications. It extracts text fields and uploaded files from incoming requests to populate the request body and file objects for server-side use. The project utilizes pluggable storage engines to abstract file persistence, allowing developers to switch between saving files to a physical disk or holding them as buffers in system memory. It supports customizable file naming strategies and a custom storage engine interface for defining how files are saved and deleted. The middleware incl
Holds uploaded files as buffer objects in volatile memory for immediate programmatic processing.
gnet is a high-performance event-driven networking framework for Go, designed for building scalable TCP, UDP, and Unix socket servers. It functions as a non-blocking socket manager and multi-reactor network engine that handles thousands of simultaneous connections with low memory overhead. The framework distinguishes itself by using a multi-reactor architecture that distributes I/O across multiple event loops pinned to operating system threads to minimize context switching. It employs edge-triggered polling to reduce system call frequency and utilizes elastic ring-buffers to minimize allocati
Minimizes allocation costs by employing reusable buffer structures that adapt to incoming network data.
ffmpeg-python is a Python wrapper that translates programmatic method calls into command-line arguments for executing FFmpeg media processing tasks. It functions as a multimedia transcoding interface and a media stream capture tool, allowing for the recording of live audio and video from hardware devices and network sources. The library features a fluent interface for constructing complex directed graphs of audio and video filters through method chaining. It also includes an FFprobe metadata extractor that retrieves structured technical properties from media files and returns them as Python d
Streams raw audio or video data directly into memory buffers for real-time processing by external software.
android-gif-drawable is a rendering library for displaying and controlling animated GIF images within Android image views and drawables. It provides a custom drawable implementation for frame-based animations, a playback system for seeking and looping, and a metadata extractor for retrieving technical properties such as frame counts and loop settings. The library enables the synchronization of a single animation instance across multiple views to ensure consistent playback. It supports loading GIF data from various sources, including assets, resources, URIs, byte arrays, files, and input strea
Uses specialized memory buffers for frame data to reduce garbage collection overhead during playback.
This project is a cross-platform mobile camera library used to embed live camera feeds and capture photos or videos within mobile applications. It provides a unified interface for integrating mobile image capture and camera functionality across different platforms. The library includes specialized tools for reading and decoding barcode data from a live camera stream and a system for locating and identifying human faces in real-time.
Optimizes data transfer by handling high-resolution image and video buffers in native memory.
xv6 is a Unix-like educational operating system designed for teaching kernel concepts. It is implemented as an instructional kernel for x86 and RISC-V architectures, featuring a virtual memory manager and support for symmetric multiprocessing. The system is designed to demonstrate core OS principles through a simplified re-implementation of Unix Version 6. It provides a controlled environment for studying system calls, trap handling, and process lifecycles. Its capability surface covers process management, including scheduling and context switching, and memory management via page tables for
Reads a specified number of bytes from a file descriptor into a memory buffer.
RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen
Provides controls for memory allocation during data pre-reading to balance performance and memory consumption.
npkill is a command-line filesystem scanner and disk space recovery utility. It is designed to identify and delete large dependency folders, specifically targeting node_modules directories created by the npm package manager, to reclaim storage space across multiple projects. The tool provides an interactive terminal interface that allows for the discovery and identification of dependency folders along with their total disk size and modification dates. Users can select directories for individual or bulk deletion using keyboard inputs. The utility includes capabilities for recursive filesystem
Uses memory buffering to read filesystem metadata in chunks, preventing system memory overload during scans.
Okio is a Java I/O library providing a set of tools for efficient byte-stream processing and file system operations. It functions as a buffered byte stream handler and streaming data transformer, utilizing a cross-platform file system API to manage data movement. The project is distinguished by its use of pooled mutable byte buffers that treat sequences as queues to reduce memory copying and garbage collection churn. It further decouples file operations from the host operating system through an abstraction-based file system, allowing for consistent path manipulation and atomic operations acro
Manages byte buffers and streams with pooled memory to reduce garbage collection and improve I/O performance.
LiveTalking is an interactive talking head engine and AI avatar management platform designed to synchronize synthetic speech with facial movements. It functions as a real-time orchestrator that connects large language models and text-to-speech services to neural-rendered digital humans. The project distinguishes itself through low-latency streaming capabilities and the ability to handle real-time conversational interruptions. It supports advanced audio-visual customization, including human voice cloning and the ability to drive avatar expressions using real-time webcam data. The platform cov
Uses a sliding-window memory buffer for video frames to prevent disk read bottlenecks during long-duration streaming.
Fluent Bit est un collecteur de logs et de télémétrie unifié cloud-native conçu comme un pipeline de données efficace en ressources. Il ingère des logs, des métriques et des traces provenant de multiples sources, les traitant en temps réel avant d'acheminer les données vers des backends de stockage externes. Le projet fonctionne comme un processeur de flux en temps réel et un processeur de logs OpenTelemetry, capable de transformer et de filtrer les données en utilisant SQL et une logique conditionnelle. Il agit également comme un agent de traçage distribué capable d'échantillonner les traces pour réduire le volume de données tout en préservant les chemins de requête complets. Le système fournit une livraison de données fiable grâce à une mise en mémoire tampon basée sur le système de fichiers et une logique de réessai avec état pour éviter la perte de données lors des pannes. Son architecture modulaire prend en charge des plugins d'entrée et de sortie enfichables, un routage basé sur les métadonnées et la capacité d'étendre les fonctionnalités via des bibliothèques partagées. Le logiciel peut être déployé en tant que conteneur sur différentes architectures CPU et systèmes d'exploitation.
Provides filesystem-backed buffering with checksums to ensure reliable data delivery and prevent loss during outages.
Lingbot-map is a feed-forward neural network designed for real-time 3D scene reconstruction from streaming video. It processes video frames one at a time without iterative optimization, producing dense geometry and camera poses at interactive frame rates directly from a live feed. The project distinguishes itself through its ability to maintain stable geometry and pose alignment across very long video sequences, handling thousands of frames without drift. It achieves this through a combination of coordinate grounding memory, sliding-window inference with overlapping keyframes, and a paged KV
Maintains a persistent coordinate frame across thousands of frames by aligning geometric cues with trajectory memory.
Hertz is a high-performance Go HTTP framework designed for building scalable microservices, RESTful APIs, and AI applications. It functions as a high-performance web server and a communication framework for microservices, utilizing non-blocking I/O and zero-copy memory management to handle high-concurrency traffic. The project distinguishes itself through a microservices communication toolkit that supports high-efficiency remote procedure calls via gRPC and Thrift protocols. It implements an asynchronous middleware engine based on an onion model, allowing for a pluggable request-response pipe
Uses memory blocks and reference counting to move data between the network and application without duplicating buffers.
RecordRTC est une bibliothèque JavaScript pour capturer des flux audio, vidéo et d'écran au sein du navigateur. Elle fonctionne comme un outil d'enregistrement multimédia qui permet la capture de flux multimédia et leur exportation sous forme de fichiers lisibles. La bibliothèque fournit des utilitaires spécialisés pour enregistrer des fenêtres de bureau ou des écrans entiers et capturer des animations 2D ou 3D à partir d'éléments canvas. Elle gère ces tâches en s'interfaçant avec les objets de flux multimédia du navigateur pour enregistrer l'activité sous forme de fichiers vidéo.
Implements memory buffering to stage recorded media chunks before consolidating them into a final downloadable file.