Flatbuffers

FlatBuffers is a cross-platform serialization library designed for performance-critical applications that require efficient, zero-copy data access. By organizing data in a structured binary format, it allows applications to read and write complex data structures directly from memory-mapped buffers without the need for intermediate parsing or temporary object allocation.

The project distinguishes itself through a schema-driven approach that balances high-performance access with long-term data evolution. It utilizes a unique memory layout featuring relative offsets and inline fixed-size structures, which ensures that data remains position-independent and cache-friendly. To maintain compatibility as data requirements change, the system employs a table-based format that allows for the addition or deprecation of fields without breaking existing readers. Furthermore, it supports polymorphic data through union-type tagged offsets, providing a flexible yet strictly typed mechanism for handling varied data structures.

Beyond its core serialization capabilities, the project provides a comprehensive ecosystem for cross-language interoperability. It includes a schema compiler that generates native source code for a wide range of programming languages, enabling type-safe access to binary data across disparate systems. The library also offers dynamic, schema-less serialization options for scenarios requiring greater flexibility, alongside robust tooling for data format conversion and build system integration.

The project is distributed with extensive support for major build systems and package managers, facilitating integration into diverse development environments. Its documentation and tooling support the entire lifecycle of data contract development, from initial schema definition to cross-platform deployment.

Features

Data Serialization Formats - FlatBuffers defines flexible data objects using tables that support field addition and deprecation to maintain forward and backward compatibility across different versions of your data.
Code Generators - FlatBuffers produces source code in multiple programming languages from schema definitions to simplify data serialization and deserialization across different parts of your application.
Schema Definition Languages - FlatBuffers specifies namespaces, types, enums, and metadata attributes for structured data serialization using a formal grammar that defines the syntax for your data.
Binary Serialization Formats - FlatBuffers enables reading and writing binary data buffers in Dart by generating classes from schemas and using provided accessors for high-performance field retrieval.

capnproto/capnproto

13,089View on GitHub

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo

golang/protobuf

10,074View on GitHub

This project is a Protocol Buffers implementation for Go, providing a binary serialization framework to convert native data structures into a compact binary format for efficient network transmission and storage. It functions as a language bindings generator, utilizing a compiler plugin to create Go source code from platform-neutral protocol buffer definitions. The implementation includes a JSON data mapper that transforms structured binary messages into JSON format to facilitate compatibility with web services and external APIs. It also enables cross-language data exchange by using a common s

apache/fory

4,234View on GitHub

Fory is a cross-language serialization framework and binary data serializer designed to convert complex object graphs into a compact binary format for high-performance data exchange. It includes an IDL-based schema compiler to transform interface definition language files into type-safe native data models and a schema evolution manager to maintain forward and backward compatibility. The project features a zero-copy data access layer that allows reading specific fields from binary rows without deserializing the entire object. It supports dual-mode serialization, enabling a toggle between a por

EsotericSoftware/kryo

6,529View on GitHub

Kryo is a Java binary serialization framework and object cloning library used to convert Java objects into a compact binary format and restore them from bytes. It provides a system for creating deep or shallow copies of complex object graphs without requiring binary conversion. The project utilizes variable-length integer encoding and integer-ID class registration to reduce the size of serialized data. It handles complex object graphs through circular reference tracking to prevent data duplication and infinite loops. To maintain long-term data viability, it includes schema evolution managemen

googleflatbuffers

Features