protobuf icon indicating copy to clipboard operation
protobuf copied to clipboard

[7392] [cpp] Fix IgnoreUnknownEnumStringValueInMap conformance tests

Open antongrbin opened this issue 1 year ago • 6 comments

Motivation

This PR fixes failing JSON conformance tests for cpp with name IgnoreUnknownEnumStringValueInMap. Specifically, for the following input of type TestMapOfEnums:

{
  "enum_map": {
    "key1": "PROTOCOL",
    "key2": "UNKNOWN_ENUM_STRING_VALUE",
    "key3": "BUFFER"
  }
}

The JSON parser with ignore_unknown_fields=true should produce a map with 2 entries, skipping the unknown enum value under key2. The implementation before this PR outputs key2 with 0-th enum value (PROTOCOL) instead.

The JSON parsing spec was discussed in https://github.com/protocolbuffers/protobuf/issues/7392.

Recent similar changes in other languages:

  • Python: https://github.com/protocolbuffers/protobuf/commit/86abf35ef5ee5b1004ec11bebb36d84c2ef6645e
  • Swift: https://github.com/apple/swift-protobuf/pull/1345
  • C#: https://github.com/protocolbuffers/protobuf/pull/15758

Changes

The main difficulty in implementing the fix is the fact that parser will first allocate a new map entry message and then consume the value. In this special case (unknown enum string value), we need to do it the other way around: first consume the value, then allocate the map entry if needed.

The PR initially contains 5 commits, with first 4 being noop unit tests and refactors:

  1. unit test that documents the mainline behavior (this test will be changed in the 5th commit)
  2. noop refactoring that simplifies the signature of ParseEnumFromStr
  3. noop refactoring that extract map key parsing into a separate function (we'll call it twice in 5th commit)
  4. noop refactoring that extracts parse map entry into a separate function (since it will get more logic)
  5. the commit that actually implements the fix and adjusts tests

antongrbin avatar Apr 11 '24 16:04 antongrbin

@haberman friendly ping to take a look :)

antongrbin avatar May 31 '24 09:05 antongrbin

@sbenzaquen can you help me ping @haberman through internal channels :) I bet this is a GitHub notifications recall miss.

I am reaching out to you since you recently reviewed https://github.com/protocolbuffers/protobuf/pull/16567

antongrbin avatar Jun 13 '24 15:06 antongrbin

@jskeet, since you worked with me on the linked issue https://github.com/protocolbuffers/protobuf/issues/7392, can you help me get reviewer's attention here through internal channels? I suspect that they are not seeing GitHub notifications.

I am not blocked on this, just wanted to eventually remove the failing tests that I added.

antongrbin avatar Jun 24 '24 18:06 antongrbin

I've pinged internally.

jskeet avatar Jun 25 '24 07:06 jskeet

@esrauchg Can you take a look at this?

googleberg avatar Sep 14 '24 05:09 googleberg

Sorry @antongrbin for the delays in getting someone to look at this! I can help review/push this forward here from the protobuf team.

[Edit: removed questions that I realized weren't relevant after looking closer]

esrauchg avatar Sep 16 '24 13:09 esrauchg

@esrauchg thank you for your time for the review and apologies for the delay in answering.

As discussed over internal chat, I will close this PR and create an internal change with the comments here addressed.

antongrbin avatar Dec 13 '24 15:12 antongrbin