Awesome GitHub RepositoriesZero-Copy

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo

Phalcon is a full-stack PHP web framework implemented as a compiled C extension that loads directly into the PHP interpreter. Rather than executing PHP scripts at runtime, the framework runs as a native C module, bundling routing, ORM, templating, and caching into immutable structures compiled at build time. This architecture hooks directly into PHP's internal Zend Engine API to bypass userland function call overhead, processes HTTP requests through a C-level event pipeline, and passes data between layers using pointer references instead of duplicating memory buffers. The framework manages ob

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

The Intel RealSense SDK is a software development kit providing drivers and libraries for interfacing with depth cameras to capture color, depth, and infrared data streams. It includes a depth camera driver for device discovery and sensor configuration, a stereo vision library for computing depth maps and aligning frames, and a 3D point cloud generator to transform depth and infrared frames into spatial representations. The SDK distinguishes itself through on-chip depth calculation and stereo calibration, using internal vision processors to reduce host CPU load. It supports hardware-level str

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Bigcache is a thread-safe, in-memory key-value store for Go designed to minimize garbage collection pauses. It functions as a memory-optimized cache capable of storing gigabytes of data while maintaining low latency during high-volume operations. The system avoids garbage collection overhead by managing memory through large byte arrays and manual allocation. It utilizes a circular-buffer layout to reuse space and lock-striped sharding to reduce contention during concurrent read and write access. The project provides capabilities for high-volume in-memory caching and low-latency data retrieva

Kitex is a high-performance remote procedure call framework for building distributed microservices in Go. It provides a communication system compatible with gRPC and Protobuf standards and includes a suite of tools for microservices governance. The framework features proxyless service mesh integration, which allows it to manage network traffic and governance directly without adding sidecar proxy latency. It optimizes data throughput through a high-performance communication library that minimizes memory copies. The system covers a broad surface of distributed capabilities, including service d

better-sqlite3 is a high-performance SQLite3 client for Node.js that executes queries synchronously, returning results directly without callbacks or promises. It compiles as a native addon using N-API, binding directly to the SQLite3 C library for immediate query execution and zero-copy result serialization into native JavaScript objects. The library is optimized for Write-Ahead Logging (WAL) mode, enabling faster concurrent reads and writes in web applications. It provides durability level tuning through the synchronous pragma, allowing adjustments between FULL, NORMAL, and OFF modes to bala

CXX is a code generator that produces a safe, zero-overhead FFI bridge between Rust and C++. It lets functions and data types from either language be used directly in the other, with static type safety enforced at compile time and no copying, serialization, or runtime checks across the boundary. The bridge supports the full range of cross-language interactions: Rust can call C++ functions and use C++ types, and C++ can call Rust functions and use Rust types. Standard library types like strings, vectors, and smart pointers are mapped automatically between the two languages, while opaque types

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Fluvio este o platformă distribuită de streaming de evenimente și un motor de streaming cloud-native conceput pentru colectarea, persistența și replicarea fluxurilor de date în timp real într-un cluster distribuit. Funcționează ca un pipeline de date în timp real pentru construirea fluxurilor de lucru cu stare care ingerează, îmbogățesc și exportă date între surse și destinații externe. Platforma se distinge prin utilizarea WebAssembly pentru a executa module compilate pentru transformări și filtrare de date în linie. Acest lucru permite execuția logicii de business personalizate pentru a remodela informațiile în mișcare fără a necesita o repornire a clusterului. Sistemul acoperă o gamă largă de capabilități, inclusiv ingestia de date bazată pe conectori din protocoale externe, stocare imutabilă structurată pe log-uri cu IO de tip zero-copy și scalarea orizontală a clusterului. Suportă crearea de pipeline-uri complexe bazate pe evenimente care utilizează procesarea cu stare, agregări pe ferestre și distribuția datelor bazată pe partiții. Motorul poate fi implementat ca un binar ușor pe diverse arhitecturi de sistem, inclusiv dispozitive IoT ARM64 pentru procesarea datelor la margine (edge).

This is a cross-platform media processing library that reads, writes, encodes, and decodes media in both browser and server environments. It supports common container formats including ISOBMFF, Matroska, Ogg, MPEG-TS, and HLS, and handles codec operations through a combination of WebCodecs API and WebAssembly-based encoders. Media is processed in streaming pipelines that maintain constant memory usage and automatically apply backpressure from output speed to all upstream components. The library distinguishes itself through a plugin-based codec registration system that allows extending support

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

TileLang is a Python-embedded domain-specific language compiler that JIT-compiles and autotunes GPU kernels. It uses a tile-based DSL, automatic software pipelining, and parallel autotuning to generate optimized GPU kernels at runtime. It supports tensor core operations with Pythonic syntax, automatic memory management, and thread mapping. The compiler searches over tile sizes, thread counts, and scheduling policies, compiling and benchmarking candidates in parallel to find the fastest kernel. It also caches compiled binaries and tuning results to disk for reuse across sessions. TileLang inc

flutterrustbridge is a code generation tool that automatically creates type‑safe Dart bindings for Rust functions, enabling direct cross‑language calls between Flutter and a Rust backend. It provides an async FFI adapter that transforms synchronous Rust functions into Dart async methods with background thread management, a cross‑language object manager that wraps persistent Rust structs as Dart objects preserving state across calls, and trait object interop that converts Rust traits into Dart abstract classes for seamless bidirectional use. The project handles the full lifecycle of integrat

Anchor is a framework for building, testing, and deploying Solana smart contracts. It provides a Rust domain-specific language (DSL) that uses macro‑based instruction dispatch and derive‑macro account validation to automate ownership checks, discriminator enforcement, and serialization before instruction execution. The framework also includes a CLI toolchain for scaffolding, compiling, and managing contracts across workspaces, a test harness that simulates program execution with account manipulation and compute unit benchmarking, and an IDL generator that emits a standardized JSON interface fr

Fury este un framework de serializare binară multi-limbaj conceput pentru encodarea obiectelor de domeniu și a grafurilor complexe, pentru a facilita schimbul de date între limbaje. Include un compilator de limbaj de definire a interfețelor (IDL) care traduce definițiile de schemă în tipuri native idiomatice și boilerplate de serializare în mai multe limbaje. Proiectul se distinge printr-un cititor binar zero-copy care permite accesarea unor câmpuri specifice fără a deserializa întregul obiect, precum și un serializator de grafuri de obiecte care păstrează referințele circulare și integritatea referențială. De asemenea, dispune de un convertor de date care transformă datele binare bazate pe rânduri în formate coloanare Apache Arrow pentru sarcini analitice. Framework-ul acoperă domenii largi de capabilități, inclusiv evoluția schemei bazată pe metadate pentru compatibilitate înainte și înapoi, un proces de compilare AOT (Ahead-Of-Time) pentru a elimina reflexia la runtime și deserializarea securizată prin validarea tipurilor bazată pe whitelist. Oferă, de asemenea, integrare pentru apeluri de proceduri la distanță de înaltă performanță prin gRPC.

rkyv este un framework de deserializare zero-copy pentru Rust care oferă un format de serializare binară pentru arhive de date memory-mappable. Permite structurilor de date complexe să fie mapate la bytes și accesate direct dintr-un buffer fără a aloca memorie nouă sau a copia date. Proiectul permite serializarea tipurilor polimorfice și a obiectelor de tip trait, menținându-le comportamentul dinamic și structura în forma binară. Utilizează adresarea prin pointeri relativi și împachetarea structurilor aliniate la byte pentru a se asigura că datele rămân valide indiferent de locul în care sunt încărcate în memorie. Framework-ul acoperă persistența datelor de înaltă performanță și gestionarea stării cu latență scăzută prin memory mapping direct. Oferă mecanisme atât pentru accesul la date zero-copy, cât și pentru reconstrucția completă a tipului de date atunci când este necesar de către aplicație.

redb is an embedded key-value store and ACID-compliant storage engine. It functions as a persistent storage system for saving and retrieving data as key-value pairs within a tree structure. The engine is built as an MVCC transactional database, utilizing multi-version concurrency control to manage simultaneous reads and writes without blocking. It employs a single-writer multi-reader model to ensure data consistency while allowing multiple threads to access the store. The system provides persistent state management and atomic transaction management to prevent data corruption during crashes.

liburing is a C library that provides a low-level wrapper for the Linux kernel io_uring interface. It serves as a programming interface for executing non-blocking disk and network operations and abstracts the system calls required to communicate with the Linux kernel. The library focuses on reducing system call overhead and context switching for high-throughput data processing. It implements mechanisms for shared ring buffers, zero-copy buffer registration, and fixed-file descriptor mapping to minimize internal lookup and reference counting overhead. The project covers asynchronous input and