Fory

Fory is a cross-language serialization framework and binary data serializer designed to convert complex object graphs into a compact binary format for high-performance data exchange. It includes an IDL-based schema compiler to transform interface definition language files into type-safe native data models and a schema evolution manager to maintain forward and backward compatibility.

The project features a zero-copy data access layer that allows reading specific fields from binary rows without deserializing the entire object. It supports dual-mode serialization, enabling a toggle between a portable wire mode for multi-language interoperability and a native mode optimized for single-language performance.

The framework covers a broad range of capabilities, including reference-tracking for circular dependencies, polymorphic type handling, and secure deserialization through type whitelisting and depth limiting. It further provides integration for columnar data transformation for analytics and the generation of service companions for remote procedure calls.

Native binaries can be produced via ahead-of-time compilation to reduce startup time and memory consumption.

Features

Language-Neutral Data Serialization - Converts complex data structures into a compact binary format for efficient exchange between different programming languages.
High-Performance Binary Serialization - Provides high-performance conversion of complex object graphs into compact binary formats for minimal latency.
Binary Deserialization - Reconstructs native language objects from binary streams based on a shared schema.
Cross-Language Data Protocols - Provides standardized binary layouts to ensure data compatibility across different programming languages and hardware architectures.
Schema Compatibility Validators - Matches local fields against remote metadata to ensure backward and forward compatibility during data evolution.
Data Schema Management - Defines data structures using builders or decorators to create optimized serialization functions.
Schema Versioning - Matches local fields against remote metadata identifiers to maintain forward and backward compatibility during schema evolution.
Random Field Accesses - Reads individual data fields from a binary row without deserializing the entire record.
IDL Model Generation - Produces compiler output for models and schema modules to define data structures used for serialization from IDL files.
IDL Compilers - Transforms interface definition language files into type-safe native data models and serialization code.
Zero-Copy Data Access - Enables reading specific data fields directly from binary buffers without reconstructing the entire object hierarchy.
Schema-Based Code Generators - Compiles language-independent IDL schema definitions into native data structures and type-safe serialization helpers.
Cross-Language Serialization Frameworks - Implements a system for converting object graphs into binary formats to ensure type-safe communication across different languages.
Type Mapping - Translates schema expressions and wire tags into native data types to ensure consistent cross-language data exchange.
Object Serialization - Encodes complex Java object graphs and primitive arrays into a compact binary format.
Serializer Code Generators - Produces companion files from marked classes to automate the conversion between high-level objects and binary data.
Reference Tracking - Tracks object identities during encoding to preserve shared references and resolve circular dependencies in complex object graphs.
Schema Evolution Strategies - Maintains backward and forward compatibility of data models through an IDL-based schema evolution manager.
Wire Format Interoperability - Implements a standardized binary layout and schema mapping to maintain data compatibility across different programming runtimes.
Multi-Language Model Generators - Generates native data structures and type-safe models across multiple languages from a single shared contract.
Columnar Data Processors - Implements a high-performance engine for processing and converting row-based serialized data into columnar layouts.
Kotlin Type Serialization - Converts Kotlin data classes, unsigned primitives, and ranges into a compact binary format.
Inheritance Management - Allows including fields from base classes into derived types to maintain structured data hierarchies.
Polymorphic Type Serializers - Serializes objects based on their actual runtime types to support inheritance hierarchies.
Analytics-Optimized Row Formats - Encodes data in a row format to enable high-throughput processing and zero-copy access for analytics.
Collection Serialization - Encodes collections and maps with optimized headers for element type sharing and nullability.
Temporal Data Converters - Serializes timestamps, dates, and durations using standardized epoch-based offsets and nanosecond adjustments.
Zero-Copy Deserialization - Reads binary data directly from a buffer without allocating new memory for reconstructed objects.
Native AOT Compilation - Generates native binaries using ahead-of-time compilation to reduce startup time and memory usage.
Remote Procedure Call Frameworks - Generates type-safe service definitions and serialization helpers for low-latency communication between distributed services.
Interoperability Modes - Enables switching between a portable wire mode for multi-language interoperability and a native mode for optimized performance.
Whitelisted Deserialization - Protects against unauthorized code execution by restricting object instantiation to a set of registered trusted types during decoding.
Decimal Arithmetic Types - Represents exact decimal values using a signed scale and unscaled integer to avoid binary floating-point precision errors.
Deserialization Security - Protects against unauthorized type instantiation and recursive object attacks through whitelists and depth limiting.
Enum Tagging Strategies - Implements configurable strategies for encoding enum constants using ordinals, numeric tags, or string names.
Static Serialization Engines - Generates serializers via annotation processing to enable reflection-free, compact data exchange.
Circular Reference Detection - Tracks object identities during serialization to handle circular dependencies and shared references.
Dual-Mode Serialization - Allows toggling between a portable wire mode for interoperability and a native mode optimized for single-language performance.
Deserialization Depth Limiting - Protects against recursive object attacks using depth limiting and type whitelists.
Symmetric Language Optimizations - Uses a native serialization mode to leverage language-specific type systems when producers and consumers share a language.
Serialization Hooks - Implements mechanisms for customizing serialization behavior, including object replacement and proxy handling.

google/flatbuffers

25,558View on GitHub

FlatBuffers is a cross-platform serialization library designed for performance-critical applications that require efficient, zero-copy data access. By organizing data in a structured binary format, it allows applications to read and write complex data structures directly from memory-mapped buffers without the need for intermediate parsing or temporary object allocation. The project distinguishes itself through a schema-driven approach that balances high-performance access with long-term data evolution. It utilizes a unique memory layout featuring relative offsets and inline fixed-size structu

capnproto/capnproto

13,089View on GitHub

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remo

neuecc/MessagePack-CSharp

6,710View on GitHub

MessagePack-CSharp is a high-performance binary serialization library for .NET applications that converts object graphs into the MessagePack format. It functions as a C# data serialization toolkit and a polymorphic binary encoder capable of handling abstract classes and interfaces using union keys to identify concrete derived types. The library provides a binary format transcoder to transform binary data into human-readable JSON for debugging. It supports ahead-of-time formatter generation to avoid runtime overhead and implements LZ4 binary compression to reduce the size of serialized data.

google/protobuf

71,412View on GitHub

Protocol Buffers is a binary serialization framework used to encode structured information into compact payloads to reduce network bandwidth and storage. It functions as a cross-language data interchange standard that enables different platforms and languages to exchange structured data using a shared schema. The project includes an interface definition language compiler that transforms schema definitions into type-safe source code for multiple target programming languages. This mechanism decouples data structures from specific language memory layouts and ensures consistent data handling acro

apachefory

Features