gcp-ingestion icon indicating copy to clipboard operation
gcp-ingestion copied to clipboard

Handle conflicting "valid" keys better when writing to BigQuery

Open relud opened this issue 4 years ago • 0 comments

When writing to BigQuery in payload format (tables in *_live datasets), field names are normalized to snake_case, and it's currently possible to have multiple fields that map to the same normalized value, as seen in https://github.com/mozilla/gcp-ingestion/pull/1297#discussion_r430696032.

Desired behavior: when ip and ip_ are both present the value for ip should be preserved, while ip_ should be sent to additional_properties.

Current behavior: the last successfully coerced non-null value will be preserved, and other values will only be sent to additional_properties if they failed to be coerced.

relud avatar May 26 '20 21:05 relud