Feature request: rust: message equality

[dfawley]

Rust message types do not implement PartialEq, and the protobuf crate does not provide an equality function.

Go provides https://pkg.go.dev/google.golang.org/protobuf/proto#Equal and, for tests, https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp.

Should Rust have similar capabilities?

@esrauchg

[esrauchg]

Protobuf discourages equality checks in most languages (eg we don't overload operator== in C++Proto), for reasons similar to things discussed here: s https://protobuf.dev/programming-guides/serialization-not-canonical/

The main problem is that it not possible to canonicalize unknown fields. This ends up especially harmful on Eq where it will work in production ... until the moment that it doesn't. This ends up especially problematic because we'd like the ability to do treeshaking which will strip fields/messages from the schemas which aren't referenced anywhere in a binary, doing so will end up causing the fields to be handled as unknown fields at runtime even if they were 'known' to the schema that the author wrote/compiled, and that optimization can make two things no longer Eq() if the schema-information caused it to parse two different byte sequences into the same result then losing the schema will make Eq suddenly return false for the same parsed inputs.

That said, we do currently have a proto_eq() which can be used for equality checks in tests here: https://github.com/protocolbuffers/protobuf/blob/main/rust/gtest_matchers_impl.rs#L13

Unit tests essentially never have unknown fields concerns, including treeshaking implications, which is itself a double edged sword because essentially no one does have test coverage for unknown fields even though they occur constantly in production, even in OSS users who often don't realize it is as common as it is.

I think we wouldn't want to make messages PartialEq for that reason in Rust, but If its sufficiently valuable to have a free function of fn proto_eq(msgA, msgB) -> bool with the clear disclaimers that it may have false-negatives in the face of unknown fields, there's no real technical blockers against it.

@dfawley Can you add a little bit of color for where the usage of this comes up and how valuable it is given the tradeoff context

@acozzette Any thoughts about the tradeoffs if we did offer a proto_eq() free fn?

[dfawley]

@dfawley Can you add a little bit of color for where the usage of this comes up and how valuable it is given the tradeoff context

The first example came up in a trivial case: gRPC has a demo "route guide" service that uses Point messages with Latitude/Longitude fields. The server wishes to compare whether a request's Point matches another one. It's easy to compare the two fields instead, or convert to a native type. (But I'd argue the point of proto is to act as said native type, so it'd be really unfortunate if we have to also define a second type to work around a limitation like this. Note that we also can't implement PartialEq ourselves, since we are not defining PartialEq or the Point type)

A real example is in how gRPC interacts with xDS. In Go, our xDS client uses proto equality to determine whether the data we receive from the xDS server for a given resource is equivalent to what we already have cached. If it is, then we don't update the application code watching for updates. This functionality is very important as our primary xDS server has a habit of re-sending all the configuration data every few minutes, and if we were to reset the state of the world on every update, it would cause a lot of disruption. These protobuf resources are pretty complex. As with the above example, we could convert to another format in order to compare, but the size of these resources makes that more painful. However, this is what gRPC- Java and C++ do, but IIRC they both had bugs in their own equality checks. Maybe Rust makes this less error-prone due to the type system -- I'm not sure. AFAIK we don't need any advanced features like "treat repeated field as a set". (And if we did, we could find a workaround.)

Go's library defines equality in pretty concrete terms:

• Equal reports whether two messages are equal, by recursively comparing the fields of the message.
• Bytes fields are equal if they contain identical bytes. Empty bytes (regardless of nil-ness) are considered equal.
• Floating-point fields are equal if they contain the same value. Unlike the == operator, a NaN is equal to another NaN.
• Other scalar fields are equal if they contain the same value.
• Message fields are equal if they have the same set of populated known and extension field values, and the same set of unknown fields values.
• Lists are equal if they are the same length and each corresponding element is equal.
• Maps are equal if they have the same set of keys and the corresponding value for each key is equal.

An invalid message is not equal to a valid message. An invalid message is only equal to another invalid message of the same type. An invalid message often corresponds to a nil pointer of the concrete message type. For example, (*pb.M)(nil) is not equal to &pb.M{}. If two valid messages marshal to the same bytes under deterministic serialization, then Equal is guaranteed to report true.

This seems to cover everything including unknown fields. Would something like that be possible for Rust?

[esrauchg]

So just having read it in your pasted reply, IMO the Go comment is insufficiently spooky compared to the observable behavior in the face of unknown fields; it really should more strongly call out that it extremely normal for the case to be that two fields would be Equal if the schema is known but will be !Equal with the schema for that field unknown.

It says "the same set of unknown field values", but it needs to be called out "unknown field value" is almost entirely non-canonicalizable blob of bytes and so the equality comparison of them is very lossy.

This is getting in the weeds, but there's some pretty clear 'unexpected' behaviors on unknown fields that most callers would not realize based on hearing 'set of unknown values', including:

1. Unknown map fields won't ever be sorted, and map order is often not serialized the same.

2. It cannot possibly recursively treat unknown fields as sets, because it can't know if a single unknown submessage is a bytes field or a message (they look the same). It's unfortunately misleading wording to say "set of unknown values" without calling that out because if you had just like:

message Child { int32 x = 1; int32 y = 2; }
message Parent { Child c = 1; }

{c: {x: 1, y: 2}} != {c: {x:2, y:1}} if c is unknown field; once you get 'one level down' it can implement set of unknown fields" behavior.

3. It can't know if a field supports presence or even if its potentially a repeated field, so even trivial cases like:

syntax="proto3";
message M { int32 x = 1; }

If someone writes out [x:0] this message is 100% identical as [] and so Equals() will be true if only if you know the schema. But if you don't know the schema, it can't know that the x:0 is an implicit presence field, it will go into unknown fields and Equals() will return false.

4. This equals can even return true when it would return false if you knew the schema, for example: message M { oneof { int32 x = 1; int32 y = 2; } }

If someone wrote out [x:0, y:1] and someone else wrote [y:1, x:0], if you know the schema these will compare !Equal (because only the last one would win), but under the 'treat unknown values as a set' strategy it will actually return Equal.