akkomar
akkomar
With #517 we have a general mechanism to separate "debug" pings. It would be useful for consumers of this stream to have a corresponding, filtered error topic containing only errors...
Currently [messages are scrubbed](https://github.com/mozilla/gcp-ingestion/blob/fb8d34477ad100e2be2f144be83bee6ab10bce57/ingestion-beam/src/main/java/com/mozilla/telemetry/decoder/ParsePayload.java#L134) after the [payload is parsed](https://github.com/mozilla/gcp-ingestion/blob/fb8d34477ad100e2be2f144be83bee6ab10bce57/ingestion-beam/src/main/java/com/mozilla/telemetry/decoder/ParsePayload.java#L115). While looking into `mozdata.monitoring.payload_bytes_error_structured` I noticed JSON parse exceptions for some pings that are [ignored](https://github.com/mozilla/gcp-ingestion/blob/fb8d34477ad100e2be2f144be83bee6ab10bce57/ingestion-beam/src/main/java/com/mozilla/telemetry/decoder/MessageScrubber.java#L30). Since some parts of the scrubbing...
…Amplitude job This adds support to read from views introduced in https://github.com/mozilla/bigquery-etl/pull/510 and can be used as an alternative solution if S3 ingestion takes too long to set up.
We're running Junit v4, which turns out to be clashing with log4j v2.2+ in ingestion-sink. After the update we can remove Dependabot ignores added in https://github.com/mozilla/gcp-ingestion/pull/2664
Although Java 11 is still maintained, some libraries start to drop support for this version (example: https://github.com/mozilla/gcp-ingestion/pull/2657). Per https://cloud.google.com/dataflow/docs/support/beam-runtime-support#java after https://github.com/mozilla/gcp-ingestion/pull/2658 lands we can jump to 21, or in case...