gcp-ingestion
gcp-ingestion copied to clipboard
Handle conflicting "valid" keys better when writing to BigQuery
When writing to BigQuery in payload
format (tables in *_live
datasets), field names are normalized to snake_case, and it's currently possible to have multiple fields that map to the same normalized value, as seen in https://github.com/mozilla/gcp-ingestion/pull/1297#discussion_r430696032.
Desired behavior: when ip
and ip_
are both present the value for ip
should be preserved, while ip_
should be sent to additional_properties
.
Current behavior: the last successfully coerced non-null value will be preserved, and other values will only be sent to additional_properties
if they failed to be coerced.