druid icon indicating copy to clipboard operation
druid copied to clipboard

Protobuf transform handling with default values

Open poriniki opened this issue 1 month ago • 0 comments

Affected Version

Druid 33.0.0

Description

There appears to be an issue in the Protobuf transform handling where default-valued fields in non-optional proto definitions are being set to null during ingestion.

Observed behavior:

  • When a non-optional proto field (for example, an int32 with a default value of 0) is used inside a transform expression, the resulting value becomes null instead of retaining its default value.
  • This behavior does not occur with optional fields. Optional fields behave as expected.
  • Because of this, we are forced to wrap non-optional proto fields with functions like nvl() or similar null-handling expressions to avoid unexpected nulls in the ingestion pipeline.

From code inspection:

  • I think this problem should be in convertMessage on /extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufConverter.java
  • It looks like the getAllFields() logic, when a value is set to default, treats default proto values (like 0 for int32) as equivalent to "not set", resulting in the field being mapped to null.
  • This appears inconsistent with Protobuf semantics, where default numeric values are valid values and not indicators of missing fields.

poriniki avatar Dec 01 '25 10:12 poriniki