Support for missing / deleted properties

[chylex]

Hi, the Version tolerant section mentions that

MemoryPackable objects, members can be added, but not deleted

I'm trying to use MemoryPack for RPC between two sides that should support different versions.

If I add a property to one side, serialize it, and send it to the other side which doesn't have the added property yet, deserialization crashes in MemoryPackSerializationException.ThrowInvalidPropertyCount.

It's a bit confusing, because technically I added a property, but for the deserializer it's as if I deleted it, which is not supported. Would it be possible to ignore deleted properties?

[neuecc]

Yes, for the deserializer's perspective, the properties appear to be deleted.

In an RPC scenario, the client must be updated before the server. An updated client has no problem connecting to the old server.

using MemoryPack;

// disallow versioning
{
    // serialize added property
    var bin = MemoryPackSerializer.Serialize<Version2>(new Version2 { MyProperty1 = 10, MyProperty2 = 20 });

    // other side which doesn't have the added property yet
    var v1 = MemoryPackSerializer.Deserialize<Version1>(bin); // throws exception

    Console.WriteLine(v1!.MyProperty1);
}

// allow versioning
{
    // serialize old schema
    var bin = MemoryPackSerializer.Serialize<Version1>(new Version1 { MyProperty1 = 10 });

    // other side, added property
    var v2 = MemoryPackSerializer.Deserialize<Version2>(bin);

    Console.WriteLine(v2!.MyProperty1); // 10
}

[MemoryPackable]
public partial class Version1
{
    public int MyProperty1 { get; set; }
}

[MemoryPackable]
public partial class Version2
{
    public int MyProperty1 { get; set; }
    public int MyProperty2 { get; set; }
}

[chylex]

In an RPC scenario, the client must be updated before the server.

That won't work in my case, both sides need to work with mismatched versions of the other side, and be able to deserialize their messages that might have added properties.

Are deleted properties not supported because support for them just wasn't implemented yet, or is it a principle of the project that it will never support deleted properties? If it's just about implementation, I will try to add support for them so I could make a PR, but obviously I won't make a PR if you don't want to support deleted properties.

[neuecc]

It is not a project principle, but it is impossible due to binary specifications designed backwards from performance.

[chylex]

I have an idea in mind (adding size of packed object to its header), though it won't be compatible with the MemoryPack format so it wouldn't be good for a PR. Feel free to close the issue then, unless you would like to keep it open for more discussion on this topic.

[neuecc]

Due to IBufferWriter's specifications, finalizing the size later is more difficult than you might imagine. The only way to be sure is to hoard and copy once in the buffer, but additional copies carry a performance penalty.

Since there are 250~254 reserved areas, it would be possible to add them without compromising compatibility. For example, 254 could be used as an "extension versioning" area, and the additional 5 bytes could be used for size + property count. However, added size solution is limited (it cannot handle deletion in the middle of a property). So if you can think of a better specification, I think it would be a good addition.

[chylex]

I haven't tried anything yet because both Visual Studio and Rider are having issues with .NET 7 / C# 11 on my computer, but my initial idea was:

1. Write a placeholder integer in the header and remember its position
2. Write the rest of the object
3. Calculate the difference between current position and placeholder's position
4. Write it back at the placeholder's position, that will be the size of the object

Then for deserialization:

1. Read the size from header
2. Read all known properties
3. Calculate how many bytes are remaining in the object and skip them

This would only work for non-streaming serialization with a different interface then IBufferWriter, and would only allow deleting at the end as you say. With the need to rewrite previous parts of the buffer, it would probably not be a good official solution for MemoryPack, but I can build a custom version of MemoryPack for my specific use case, where I can ensure that my buffer can be rewritten and I never delete from the middle.

I don't know what would be the best general solution that supports everything MemoryPack does. I only tried it out for the first time a few hours ago so I know very little, but maybe I will get some better ideas once I start digging in the code.

[neuecc]

I thought it would be good to add this feature, thank you. I will try to come up with ideas for best performance. One thing I'm thinking about is.

• header, record size of values(fixed-int)
• footer, array[memberCount] of offset delta with varint

The fact that the headers are fixed-int helps with optimization. Recording the offset of each member allows skip. This can be varint + delta to reduce size. If the size is variable-length, place it in the footer so that it does not interfere with optimization.

[neuecc]

I've released v1.5.0, it supports full-version-tolerant. improving performance is the next.

[chylex]

Awesome, thanks!