Custom Protobuf Options / Schema Registry Support

[trickleup]

Description

The confluent schema registry (java) recently added support for handling of custom protobuf options. What that meant was that both declarations of custom options, as well as invocations of them, would now get picked up by the registry, whereas before they were just ignored and filtered out:

// Declaration
extend .google.protobuf.MessageOptions {
  bool my_custom_option = 50000;
}

// Invocation
message MyMessage {
  option (my_custom_option) = true;
}

However, confluent-kafka-python has not kept up with this change. When producing a message using the ProtobufSerializer (here), it uses the _schema_to_str method which looks up a schema id from the registry by sending a base64 encoded FileDescriptor to match it. This method of matching a schema via the schema registry client's lookup_schema method works with declarations of custom options, but does not include invocations of them, per above.

That means, that if there are protobuf schemas in the registry with custom options invocations, the python producer will not be able to match those, and will throw a 404 error, and not be able to get a schema id for the produce call.

In the confluent java client, this is not an issue, I believe it's handled here.

Are there plans to fix this in confluent-kafka-python (and/or the python protoc plugin, not sure where the actual issue is)?

Are there any known workarounds to get this to work prior to such a fix?

Thanks - let me know if I misunderstood something or if I'm looking at this the wrong way.

How to reproduce

1. Create a protobuf schema with a custom option in it.
2. Register the schema with the registry as a plain string.
3. The schema registry will now have a correct representation of the schema with a custom option in it.
4. Produce a message using that schema using the ProtobufSerializer, setting auto.register.schemas to false
5. Producing of the message will fail with a "schema not found" error from the registry.

Checklist

• confluent-kafka-python version: 1.9.2
• Running against confluent cloud 7.3