gcp-ingestion icon indicating copy to clipboard operation
gcp-ingestion copied to clipboard

Consider some message scrubbing before parsing JSON payloads

Open akkomar opened this issue 2 years ago • 0 comments

Currently messages are scrubbed after the payload is parsed. While looking into mozdata.monitoring.payload_bytes_error_structured I noticed JSON parse exceptions for some pings that are ignored. Since some parts of the scrubbing process do not require message to be parsed (e.g. we have document namespace and type available beforehand), we could split and run them before attempting to parse to avoid doing some unnecessary work.

akkomar avatar Jan 03 '23 14:01 akkomar