[OSPP] [Rust] Implementing Type Forward/Backward Compatible Serialization For Fury Rust

[chaokunyang]

Feature Request

(1) Background: Fury is a high-performance serialization library that supports cross-language binary protocols. Rust-based applications are increasingly growing. Rust is widely used for its memory safety and concurrency performance, but it also faces serialization challenges related to compatibility with different versions and type structures. Ensuring forward and backward compatibility of types is crucial for long-term project evolution and multi-version support in Rust serialization.

(2) Existing Work: Fury has already achieved efficient serialization and deserialization in languages such as Java, Python, and NodeJS. However, in Rust, the related work is still at an early stage, only supporting serialization and deserialization when object type schemas are strictly consistent. As projects expand and become more complex, serialization solutions need to be more comprehensive and efficient to support version evolution.

(3) Existing Shortcomings: The current Fury Rust serialization mechanism has yet to fully support forward and backward compatibility of structure types. There is a lack of adaptation mechanisms to handle added or removed fields during type updates, ensuring the stability and consistency of serialized data. Also, efficient sharing of metadata has not yet become a best practice.

(4) Points for Improvement: There is a need to implement a type compatibility mechanism based on shared metadata, allowing Fury Rust serialization to support dynamic type updates and ensure compatibility between different versions. Through efficient metadata sharing, intelligent serialization processing for added, modified, or removed fields should be achieved to reduce the development burden and operational risk associated with version changes.

(5) Final Project Goals: The ultimate goal is to implement a serialization framework for Fury Rust that can support forward and backward compatibility of types. This framework will leverage the metadata sharing mechanism of Fury's binary serialization protocol, with implementation references from Fury Java/NodeJS, to automatically adapt to changes in type structures. It aims to ensure that data between different versions can be correctly serialized and deserialized, enhancing the breadth and depth of Fury's application in the Rust ecosystem.

Is your feature request related to a problem? Please describe

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

This issue is open for 开源之夏OSPP

[urlyy]

I want to participate in this year's OSPP and I'm also quite interested in this issue. Could you provide some specific examples regarding this problem? I want to use examples to better locate the relevant source code and think about potential solutions.

[chaokunyang]

Hi @urlyy , thanks for the interest about this issue. We haven't prepare examples for this issue in rust. Currently you can take java/fury-core/src/test/java/org/apache/fury/serializer/MetaSharedCompatibleTest.java for an example.

The java/fury-core/src/main/java/org/apache/fury/meta/ClassDef.java and java/fury-core/src/main/java/org/apache/fury/resolver/MetaContext.java is the core class for meta share.

For document, you can refer to https://fury.apache.org/docs/specification/fury_xlang_serialization_spec#schema-evolution

[urlyy]

@chaokunyang Hi, I have some confusion about the document during implementating Extended Type Meta with Inheritance support. I'm not sure if the meta header is actually the global binary header.

If the meta header is indeed the meta header, then the position of the num_class field seems incorrect. If the meta header is the global binary header, then the fields originally in the "global binary header" would be missing. I'm confused about this.

In addition, based on my understanding, layer represents the inheritance hierarchy level, so 3 layers means the current class includes two parent classes (making num_class equal to 3). Is that correct?

[chaokunyang]

You can ignore Extended Type Meta with Inheritance support. It's only for language supports inheritance and it's not cross language compatible

[urlyy]

I noticed that this part of the logic is not reflected in the specification document, which confused me for a while. I suggest that we should add a description here.

[urlyy]

@chaokunyang Should it be 0b1111_1111_1111 here? The documentation seems to only use 11 bits.