Awesome GitHub RepositoriesBinary Serialization Formats

Protocol Buffers is a binary serialization framework used to encode structured information into compact payloads to reduce network bandwidth and storage. It functions as a cross-language data interchange standard that enables different platforms and languages to exchange structured data using a shared schema. The project includes an interface definition language compiler that transforms schema definitions into type-safe source code for multiple target programming languages. This mechanism decouples data structures from specific language memory layouts and ensures consistent data handling acro

This project is an Android messaging application and a client implementation of the MTProto protocol. It serves as a native mobile interface for real-time communication and file sharing on the Android operating system. The application functions as a client that connects to the Telegram MTProto protocol to facilitate the transmission of encrypted text and media messages between users. The system includes capabilities for secure mobile communication, messaging API integration, and the implementation of client-side logic for connecting to a centralized messaging backend.

FlatBuffers is a cross-platform serialization library designed for performance-critical applications that require efficient, zero-copy data access. By organizing data in a structured binary format, it allows applications to read and write complex data structures directly from memory-mapped buffers without the need for intermediate parsing or temporary object allocation. The project distinguishes itself through a schema-driven approach that balances high-performance access with long-term data evolution. It utilizes a unique memory layout featuring relative offsets and inline fixed-size structu

Vegeta is an HTTP load testing tool and library designed to measure the performance and stability of web services. It functions as a command-line utility, a programmable package for integration into other applications, and a distributed load generator capable of splitting request rates across multiple machines. The tool is distinguished by its constant-rate request scheduler, which dispatches requests at a fixed frequency regardless of target response times. It employs lazy target streaming to maintain low memory usage during large tests and uses a binary-encoded storage format to minimize di

Yjs is a CRDT framework and real-time state synchronization engine providing shared data types for collaborative software. It functions as a shared data type library of collaborative maps, arrays, and text structures that converge automatically across distributed clients, serving as a local-first data store and a collaborative editor backend. The system distinguishes itself through a synchronization engine that encodes document changes into commutative binary updates. It employs state-vector based differencing to transmit only missing incremental updates between peers and uses relative-positi

This project serves as a comprehensive technical reference and educational platform for the Ethereum ecosystem. It provides a deep dive into the fundamental architecture of decentralized ledger systems, covering the core mechanisms that enable trustless state transitions, cryptographic security, and network consensus. The documentation distinguishes itself by bridging high-level conceptual frameworks with practical implementation details. It details the lifecycle of smart contract development, from source code compilation and bytecode analysis to deployment and interaction patterns. Furthermo

This project is an educational framework designed to teach the fundamentals of building core distributed systems and web services from scratch in Go. It provides a collection of modular implementations that demonstrate how to construct essential infrastructure components, including web servers, remote procedure call systems, distributed caches, and database abstraction layers. The framework distinguishes itself by focusing on the internal mechanics of these systems rather than providing a high-level abstraction for production use. It covers the implementation of complex architectural patterns

ggml is a low-level C++ tensor library and machine learning inference engine designed for performing mathematical operations on multi-dimensional arrays across diverse hardware platforms. It provides a foundational toolset for executing machine learning models and calculating mathematical gradients through an automatic differentiation library. The project features a quantized tensor framework that converts floating-point weights into integer representations to reduce memory usage and increase inference speed. It utilizes a custom binary format for model serialization to ensure rapid loading a

Skynet is a distributed game server framework designed for building scalable online game backends. It utilizes distributed actor-based clusters and real-time network communication to manage high-concurrency session coordination across multiple nodes. The framework includes a cluster management orchestrator for coordinating services via cluster-wide messaging and dynamic configuration updates. It features a multi-protocol network gateway supporting TCP, UDP, and WebSockets, alongside a data encoding layer using BSON and Sproto serialization for efficient information transfer between distribute

This project is a Node.js client for PostgreSQL databases, providing a protocol parser to translate raw binary streams into JavaScript objects. It serves as a driver for executing queries, managing data, and integrating Node.js applications with PostgreSQL backends. The library includes a connection pool manager to reduce network overhead by caching reusable connections and a result streamer that uses cursors to retrieve large datasets incrementally. It also functions as an event listener for subscribing to asynchronous server-side notifications to trigger real-time application events. Broad

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo

Hazel is a C++ game development framework and 3D graphics rendering engine designed for building high-performance interactive software. It provides a foundational architecture that enables the construction of desktop applications through native hardware acceleration. The project includes a dedicated game engine editor that allows for the design and testing of interactive 3D and 2D scenes within a unified environment. This editor utilizes an immediate mode interface to maintain synchronization with the engine state, while a hardware-agnostic abstraction layer handles rendering commands across

Garnet is a multi-threaded in-memory database and distributed key-value store. It functions as a high-performance remote cache store that implements the RESP wire protocol to maintain compatibility with existing Redis clients and libraries. The project is distinguished by a shared-memory architecture that enables parallel request processing across multiple cores for sub-millisecond latency. It features a tiered storage system that automatically offloads colder data from system memory to SSD or cloud storage layers, and includes a specialized vector search database for high-dimensional similar

Pomelo is a distributed Node.js game server framework designed for building scalable multiplayer game backends and real-time web applications. It employs a multi-process architecture to distribute application logic horizontally across a cluster of machines, enabling high concurrency and balanced network traffic. The framework utilizes a WebSocket communication layer to manage bidirectional data exchange between diverse web clients and server clusters. It incorporates a remote procedure call system for triggering functions across remote server processes and uses Protocol Buffers for binary ser

Thrift is a cross-language remote procedure call framework and data serialization protocol. It provides an interface definition language to specify data types and service interfaces in a neutral format, enabling the automated generation of client and server code across multiple programming languages. The project functions as a polyglot service communicator using a layered software stack to ensure interoperable communication. It focuses on implementing cross-language remote procedure calls and transforming complex data structures into standardized formats for efficient network transport. The

RenderDoc is a cross-platform graphics frame debugger and GPU profiler used to capture and analyze 3D graphics API calls and state for a single frame of rendering. It functions as a debugging tool for Vulkan, OpenGL, OpenGL ES, and Direct3D 11 and 12. The tool provides specialized debugging capabilities for inspecting graphics pipelines and resource states across Windows, Linux, Android, and Nintendo Switch. It enables the analysis of DirectX shaders, OpenGL application profiling, and Vulkan API development to verify rendering behavior and fix visual bugs.

protobuf.js is a JavaScript and TypeScript library for encoding and decoding structured data using the Protocol Buffers binary format. It functions as a dynamic parser that can load and process schemas at runtime, a JSON transcoder for converting messages to human-readable formats, and a framework for building transport-agnostic remote procedure call clients. The project distinguishes itself by offering both dynamic message handling and a static code generator that produces TypeScript declarations and JavaScript modules to reduce runtime overhead. It also provides a reflection API for definin

This project is a framework for the efficient serialization and deserialization of data structures. It provides a unified, macro-based interface that automates the conversion of complex internal objects into standardized formats and reconstructs them from raw input streams or buffers. By leveraging compile-time code generation, the library minimizes manual implementation overhead while ensuring consistent logic across diverse data types. The framework distinguishes itself through a format-agnostic data model and a visitor-based parsing architecture that decouples data structures from specific

The MongoDB Node.js Driver is a programmatic interface and NoSQL database client used to manage document storage and execute operations within a MongoDB database. It serves as an asynchronous database interface and connection manager that enables Node.js applications to integrate with MongoDB servers. The project implements client-side field encryption to secure sensitive data and queries locally before transmission. It also provides a BSON serialization library to convert JavaScript objects into a binary format for efficient storage and network transmission. The driver covers a broad range

phpredis is a C-based native extension that bridges PHP applications with Redis servers for high-performance data storage and retrieval. It serves as an interface for manipulating strings, hashes, lists, sets, and sorted sets while providing a direct path for executing Redis commands and server-side scripts. The extension provides comprehensive support for distributed environments and high availability. It interfaces with Redis Cluster to distribute data across multiple nodes using hash slots and manages Redis Sentinel for service discovery and automatic failover. It also enables shared state