Awesome GitHub RepositoriesZero-Copy

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo

Phalcon is a full-stack PHP web framework implemented as a compiled C extension that loads directly into the PHP interpreter. Rather than executing PHP scripts at runtime, the framework runs as a native C module, bundling routing, ORM, templating, and caching into immutable structures compiled at build time. This architecture hooks directly into PHP's internal Zend Engine API to bypass userland function call overhead, processes HTTP requests through a C-level event pipeline, and passes data between layers using pointer references instead of duplicating memory buffers. The framework manages ob

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

The Intel RealSense SDK is a software development kit providing drivers and libraries for interfacing with depth cameras to capture color, depth, and infrared data streams. It includes a depth camera driver for device discovery and sensor configuration, a stereo vision library for computing depth maps and aligning frames, and a 3D point cloud generator to transform depth and infrared frames into spatial representations. The SDK distinguishes itself through on-chip depth calculation and stereo calibration, using internal vision processors to reduce host CPU load. It supports hardware-level str

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Bigcache is a thread-safe, in-memory key-value store for Go designed to minimize garbage collection pauses. It functions as a memory-optimized cache capable of storing gigabytes of data while maintaining low latency during high-volume operations. The system avoids garbage collection overhead by managing memory through large byte arrays and manual allocation. It utilizes a circular-buffer layout to reuse space and lock-striped sharding to reduce contention during concurrent read and write access. The project provides capabilities for high-volume in-memory caching and low-latency data retrieva

Kitex is a high-performance remote procedure call framework for building distributed microservices in Go. It provides a communication system compatible with gRPC and Protobuf standards and includes a suite of tools for microservices governance. The framework features proxyless service mesh integration, which allows it to manage network traffic and governance directly without adding sidecar proxy latency. It optimizes data throughput through a high-performance communication library that minimizes memory copies. The system covers a broad surface of distributed capabilities, including service d

better-sqlite3 is a high-performance SQLite3 client for Node.js that executes queries synchronously, returning results directly without callbacks or promises. It compiles as a native addon using N-API, binding directly to the SQLite3 C library for immediate query execution and zero-copy result serialization into native JavaScript objects. The library is optimized for Write-Ahead Logging (WAL) mode, enabling faster concurrent reads and writes in web applications. It provides durability level tuning through the synchronous pragma, allowing adjustments between FULL, NORMAL, and OFF modes to bala

CXX is a code generator that produces a safe, zero-overhead FFI bridge between Rust and C++. It lets functions and data types from either language be used directly in the other, with static type safety enforced at compile time and no copying, serialization, or runtime checks across the boundary. The bridge supports the full range of cross-language interactions: Rust can call C++ functions and use C++ types, and C++ can call Rust functions and use Rust types. Standard library types like strings, vectors, and smart pointers are mapped automatically between the two languages, while opaque types

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Fluvio est une plateforme de streaming d'événements distribuée et un moteur de streaming cloud-native conçu pour collecter, persister et répliquer des flux de données en temps réel à travers un cluster distribué. Il fonctionne comme un pipeline de données temps réel pour construire des workflows avec état qui ingèrent, enrichissent et exportent des données entre des sources et des destinations externes. La plateforme se distingue par son utilisation de WebAssembly pour exécuter des modules compilés pour des transformations et filtrages de données en ligne. Cela permet l'exécution d'une logique métier personnalisée pour remodeler l'information en mouvement sans nécessiter de redémarrage du cluster. Le système couvre un large éventail de capacités, incluant l'ingestion de données basée sur des connecteurs depuis des protocoles externes, un stockage immuable structuré en logs avec E/S zéro-copie, et une mise à l'échelle horizontale du cluster. Il prend en charge la création de pipelines complexes pilotés par les événements qui utilisent le traitement avec état, les agrégations par fenêtrage et la distribution de données basée sur les partitions. Le moteur peut être déployé comme un binaire léger sur diverses architectures système, y compris des appareils IoT ARM64 pour le traitement de données en périphérie (edge).

This is a cross-platform media processing library that reads, writes, encodes, and decodes media in both browser and server environments. It supports common container formats including ISOBMFF, Matroska, Ogg, MPEG-TS, and HLS, and handles codec operations through a combination of WebCodecs API and WebAssembly-based encoders. Media is processed in streaming pipelines that maintain constant memory usage and automatically apply backpressure from output speed to all upstream components. The library distinguishes itself through a plugin-based codec registration system that allows extending support

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

TileLang is a Python-embedded domain-specific language compiler that JIT-compiles and autotunes GPU kernels. It uses a tile-based DSL, automatic software pipelining, and parallel autotuning to generate optimized GPU kernels at runtime. It supports tensor core operations with Pythonic syntax, automatic memory management, and thread mapping. The compiler searches over tile sizes, thread counts, and scheduling policies, compiling and benchmarking candidates in parallel to find the fastest kernel. It also caches compiled binaries and tuning results to disk for reuse across sessions. TileLang inc

flutterrustbridge is a code generation tool that automatically creates type‑safe Dart bindings for Rust functions, enabling direct cross‑language calls between Flutter and a Rust backend. It provides an async FFI adapter that transforms synchronous Rust functions into Dart async methods with background thread management, a cross‑language object manager that wraps persistent Rust structs as Dart objects preserving state across calls, and trait object interop that converts Rust traits into Dart abstract classes for seamless bidirectional use. The project handles the full lifecycle of integrat

Anchor is a framework for building, testing, and deploying Solana smart contracts. It provides a Rust domain-specific language (DSL) that uses macro‑based instruction dispatch and derive‑macro account validation to automate ownership checks, discriminator enforcement, and serialization before instruction execution. The framework also includes a CLI toolchain for scaffolding, compiling, and managing contracts across workspaces, a test harness that simulates program execution with account manipulation and compute unit benchmarking, and an IDL generator that emits a standardized JSON interface fr

Fury est un framework de sérialisation binaire multi-langage conçu pour encoder des objets de domaine et des graphes complexes afin de faciliter l'échange de données entre langages. Il inclut un compilateur de langage de définition d'interface (IDL) qui traduit les définitions de schéma en types natifs idiomatiques et en code de sérialisation répétitif à travers plusieurs langages. Le projet se distingue par un lecteur binaire zero-copy qui permet d'accéder à des champs spécifiques sans désérialiser l'objet entier, ainsi qu'un sérialiseur de graphe d'objets qui préserve les références circulaires et l'intégrité référentielle. Il dispose également d'un convertisseur de données qui transforme les données binaires basées sur des lignes en formats Apache Arrow colonnaires pour les charges de travail analytiques. Le framework couvre de vastes domaines de capacités, incluant l'évolution de schéma pilotée par métadonnées pour la compatibilité ascendante et descendante, un processus de compilation AOT au moment du build pour éliminer la réflexion à l'exécution, et une désérialisation sécurisée via une validation de type basée sur liste blanche. Il fournit en outre une intégration pour des appels de procédure à distance haute performance via gRPC.

rkyv est un framework de désérialisation zero-copy pour Rust qui fournit un format de sérialisation binaire pour les archives de données mappables en mémoire. Il permet aux structures de données complexes d'être mappées en octets et accédées directement depuis un tampon sans allouer de nouvelle mémoire ou copier des données. Le projet permet la sérialisation de types polymorphes et d'objets trait, maintenant leur comportement dynamique et leur structure sous forme binaire. Il utilise l'adressage par pointeur relatif et le packing de structure aligné sur les octets pour garantir que les données restent valides quel que soit l'endroit où elles sont chargées en mémoire. Le framework couvre la persistance de données haute performance et la gestion d'état à faible latence via le mappage mémoire direct. Il fournit des mécanismes à la fois pour l'accès aux données zero-copy et la reconstruction complète des types de données lorsque requis par l'application.

redb is an embedded key-value store and ACID-compliant storage engine. It functions as a persistent storage system for saving and retrieving data as key-value pairs within a tree structure. The engine is built as an MVCC transactional database, utilizing multi-version concurrency control to manage simultaneous reads and writes without blocking. It employs a single-writer multi-reader model to ensure data consistency while allowing multiple threads to access the store. The system provides persistent state management and atomic transaction management to prevent data corruption during crashes.

liburing is a C library that provides a low-level wrapper for the Linux kernel io_uring interface. It serves as a programming interface for executing non-blocking disk and network operations and abstracts the system calls required to communicate with the Linux kernel. The library focuses on reducing system call overhead and context switching for high-throughput data processing. It implements mechanisms for shared ring buffers, zero-copy buffer registration, and fixed-file descriptor mapping to minimize internal lookup and reference counting overhead. The project covers asynchronous input and