36 مستودعات
Techniques for accessing serialized data structures directly from memory without requiring deserialization or copying.
Distinguishing note: None of the candidates matched; this is a specific performance optimization for data access.
Explore 36 awesome GitHub repositories matching data & databases · Zero-Copy Data Access. Refine with filters or upvote what's useful.
نواة Linux هي نواة نظام تشغيل متجانسة تدير موارد الأجهزة، والذاكرة، وجدولة العمليات عبر بنيات حوسبة متنوعة. توفر بيئة قياسية متوافقة مع POSIX لتنفيذ التطبيقات مع الحفاظ على إطار عمل تعريف (driver framework) معياري يسمح بالتحميل والإزالة الديناميكية لواجهات الأجهزة. يتميز المشروع بمجموعة أدوات التزامن عالية الأداء، والتي تستخدم بدائيات المزامنة الخالية من القفل وآليات القراءة-النسخ-التحديث لإدارة الوصول إلى البيانات المشتركة في بيئات متعددة النواة. يتضمن مجموعة شاملة لتتبع النواة والأدوات التي تتيح مراقبة غير تدخلية لأحداث النظام، وتنفيذ الوظائف، ومقاييس زمن الوصول. علاوة على ذلك، تفرض النواة ضمانات صارمة لاستقرار الواجهة وتتبع دورة الحياة لضمان التوافق مع الإصدارات السابقة للتطبيقات التابعة. بعيداً عن هويتها الأساسية، يتضمن النظام قدرات واسعة لتجريد الأجهزة، وتنفيذ بروتوكولات الشبكة، وفرض سياسات الأمن. وهو يدعم المتطلبات الهندسية المتخصصة من خلال إدارة حالة الطاقة، وتحسينات الأنظمة المدمجة، وعمليات التمهيد القائمة على البرامج الثابتة. تتميز البنية أيضاً بأطر تشخيصية قوية لتحليل الذاكرة، والتحقق من تنفيذ النظام، والتحقق من صحة نماذج البرمجة المتزامنة. يوفر مستودع المصدر نظام بناء كاملاً لتحويل الكود إلى صور ثنائية قابلة للتنفيذ، بما في ذلك أدوات لاختيار ميزات النواة وتحسين التكوين لتكييف المخرجات مع متطلبات الأجهزة المحددة.
Uses synchronization primitives to coordinate concurrent access and prevent data conflicts.
Python is a high-level, interpreted programming language designed for readability and versatility. It operates via a bytecode-based virtual machine and manages memory automatically through reference-counting garbage collection. The language supports multiple programming paradigms, including object-oriented, imperative, and functional styles, and provides a comprehensive standard library for system operations, networking, and data handling. The language is distinguished by its dynamic nature, allowing for runtime object introspection and metaclass-driven class creation. It utilizes protocol-ba
Python retrieves specific values from a collection using zero-based integer indexing to pinpoint data at known positions.
FlatBuffers is a cross-platform serialization library designed for performance-critical applications that require efficient, zero-copy data access. By organizing data in a structured binary format, it allows applications to read and write complex data structures directly from memory-mapped buffers without the need for intermediate parsing or temporary object allocation. The project distinguishes itself through a schema-driven approach that balances high-performance access with long-term data evolution. It utilizes a unique memory layout featuring relative offsets and inline fixed-size structu
Storing and retrieving complex data structures with minimal memory overhead and zero-copy access for performance-critical applications.
simdjson is a high-performance, header-only C++ library designed for parsing, querying, and serializing JSON data with minimal memory overhead. It functions as a hardware-aware data processing engine that leverages vector instructions to achieve gigabyte-per-second parsing speeds. By detecting host processor capabilities at runtime, the library automatically selects the most efficient instruction sets to accelerate structural analysis and validation. The library distinguishes itself through a focus on extreme efficiency and resource management. It utilizes memory mapping and padded buffer ali
Enables on-demand navigation and value retrieval from JSON structures without requiring a full initial parse of the entire document into memory.
This project is an educational framework designed to teach the fundamentals of building core distributed systems and web services from scratch in Go. It provides a collection of modular implementations that demonstrate how to construct essential infrastructure components, including web servers, remote procedure call systems, distributed caches, and database abstraction layers. The framework distinguishes itself by focusing on the internal mechanics of these systems rather than providing a high-level abstraction for production use. It covers the implementation of complex architectural patterns
Improves performance in read-intensive scenarios by allowing multiple concurrent readers while maintaining exclusive access for writers.
Sass is a stylesheet compilation engine and CSS preprocessor that extends standard CSS with variables, nested rules, mixins, and functions. It functions as a comprehensive design system tool, enabling developers to organize complex stylesheets into modular, reusable components while automating the transformation of advanced syntax into browser-compatible CSS. The project distinguishes itself through its sophisticated build automation and language-level extensibility. It provides robust support for programmatic style generation, including conditional logic, iterative loops, and unit-aware math
Enables character access within strings using one-based positive or negative indexing.
Tracy is a real-time performance profiling framework for C and C++ applications. It provides a software instrumentation library that captures high-resolution telemetry data, which is then visualized through a separate graphical interface to identify bottlenecks and resource allocation issues. The system utilizes a client-server architecture that enables remote profiling, allowing performance data to be captured on a target machine and analyzed on a workstation. It employs lock-free event logging and shared-memory ring buffers to minimize the overhead of data collection, ensuring that the main
Packs binary data into compact structures for efficient transmission without expensive serialization.
CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo
Maps binary data directly to memory structures to eliminate encoding and decoding overhead.
nom is a parser combinator framework for Rust used to build complex parsers by combining small, reusable parsing functions. It functions as a zero-copy parsing tool that minimizes memory overhead by returning slices of the original input instead of allocating new memory. The framework is designed for diverse data formats, serving as a binary data parser with configurable endianness and a bitstream processing library capable of extracting values of arbitrary bit length. It also functions as a streaming data parser that can process data arriving in chunks and signal when additional input is req
Parses large inputs by returning slices of the original data to avoid memory allocation.
nom is a Rust parser combinator framework used to build complex parsers for binary and text data. It functions as an abstract syntax tree generator and a bit-level binary parser, allowing users to construct structured data by combining small, reusable parsing functions. The framework provides specialized support for zero-copy binary parsing, extracting data as slices from raw byte arrays to avoid memory allocations. It also includes a streaming data parser capable of processing partial input chunks from networks or files and signaling when additional input is required. The project covers a b
Retrieves subsets of input data as slices to eliminate memory allocation during parsing.
This project is a framework for the efficient serialization and deserialization of data structures. It provides a unified, macro-based interface that automates the conversion of complex internal objects into standardized formats and reconstructs them from raw input streams or buffers. By leveraging compile-time code generation, the library minimizes manual implementation overhead while ensuring consistent logic across diverse data types. The framework distinguishes itself through a format-agnostic data model and a visitor-based parsing architecture that decouples data structures from specific
Supports zero-copy data borrowing to map input data directly to memory references, avoiding unnecessary allocations.
OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and
Uses shared memory and zero-copy tensors to avoid expensive data duplication during inference.
cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p
Enables high-efficiency zero-copy data transfers between GPU dataframes and Apache Arrow or NumPy arrays.
LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters
Implements random-access indexing and zero-copy reads to feed data batches directly into training loops without overhead.
Sled is an embedded key-value store and ACID-compliant database designed for high-performance data persistence. It functions as a log-structured storage engine that organizes data using B+ trees to support efficient range queries and prefix scans. The engine implements a zero-copy data store model, utilizing epoch-based reclamation to provide direct references to cached values without memory allocations. It distinguishes itself through a combination of write-ahead logging, page cache optimizations to reduce write amplification on flash storage, and serializable transactions for atomic multi-k
Provides zero-copy data access by returning direct references to cached values using epoch-based reclamation.
Hertz is a high-performance Go HTTP framework designed for building scalable microservices, RESTful APIs, and AI applications. It functions as a high-performance web server and a communication framework for microservices, utilizing non-blocking I/O and zero-copy memory management to handle high-concurrency traffic. The project distinguishes itself through a microservices communication toolkit that supports high-efficiency remote procedure calls via gRPC and Thrift protocols. It implements an asynchronous middleware engine based on an onion model, allowing for a pluggable request-response pipe
Uses zero-copy APIs to send and receive data between processes and the kernel, avoiding memory buffer duplication.
The Rust RFCs repository is the formal home for the Rust language evolution process, housing the structured design documents and community review mechanisms that govern changes to the Rust programming language, its compiler, and its standard library. It defines the complete lifecycle for proposing, discussing, and implementing substantial changes through RFC documents, from initial submission and community feedback through final comment periods and sub-team sign-offs. The repository codifies the governance and collaboration processes that shape Rust's development, including mechanisms for com
Defines the copy method for transferring data between I/O streams in Rust's standard library.
HAProxy is a high-performance TCP and HTTP proxy that distributes traffic across multiple backend servers to ensure availability and fault tolerance for critical services. It operates in either TCP or HTTP mode, with an event-driven, single-threaded reactor that handles tens of thousands of connections without context switching, and supports kernel-level data transfer to minimize memory usage and latency. What distinguishes HAProxy is its configuration-file-first design, where all load-balancing rules and runtime behavior are defined in a declarative text file parsed at startup. It embeds a L
Transfers data between client and server connections directly at the kernel level to reduce memory usage and latency.
Masuit.Tools is a comprehensive static utility library for .NET and ASP.NET Core development. It provides a broad collection of reusable helper methods and infrastructure components that cover common programming tasks without requiring dependency injection or instance management. The library is organized as flat utility classes, making its functionality directly accessible from anywhere in a project. The toolkit distinguishes itself through a wide range of integrated capabilities that go beyond typical utility libraries. It includes a multithreaded range-request file downloader with pause and
Reads, writes, and computes MD5 or SHA1 for streams and large files asynchronously.
CppGuide is a curated collection of educational resources and practical guides focused on C++ server development, Linux kernel internals, concurrent programming, network protocols, and security exploitation. It provides structured learning paths for backend developers, covering everything from interview preparation to building high-performance network servers and understanding operating system fundamentals. The guide distinguishes itself by offering in-depth, hands-on tutorials that walk through real-world implementations, including building a Redis-like server from scratch, designing custom
Teaches read-copy-update synchronization for lock-free concurrent reads, a key kernel concurrency mechanism.