gcp-ingestion
gcp-ingestion copied to clipboard
mozilla
Documentation and implementation of telemetry ingestion on Google Cloud Platform
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.2 to 7.1.3. Release notes Sourced from pytest's releases. 7.1.3 pytest 7.1.3 (2022-08-31) Bug Fixes #10060: When running with --pdb, TestCase.tearDown is no longer called for tests...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
We should include at least some brief documentation about flush manager in https://mozilla.github.io/gcp-ingestion/ingestion-edge/
ignore this issue. just testing if the one way sync is working.
Background https://github.com/mozilla/gcp-ingestion/pull/732 https://app.circleci.com/settings/project/github/mozilla/gcp-ingestion/ssh?return-to=https%3A%2F%2Fapp.circleci.com%2Fpipelines%2Fgithub%2Fmozilla%2Fgcp-ingestion%3Ffilter%3Dall It looks like (from a :jason screenshot) this deploy key is possibly associated with :wlach, and is separate from the main deploy key. I'm not sure what...
With python-jwcrypto 0.9.1-2. This script should be refactored to produce the same output with non-deprecated functions. It's only used by ops logic to generate keypairs.
One known failure mode for ingestion-beam is rate limiting from the BQ API when we list datasets/tables in order to check whether destination tables exist. See https://mozilla-hub.atlassian.net/browse/DSRE-194 and https://github.com/mozilla/bigquery-backfill/pull/15 Currently,...
In `ParseUri` we don't normalize the doctype and namespace names. Instead, we rely on `SchemaStore` to normalize those attributes when looking up schemas. This means that the `metadata.uri.document_namespace` field in...
We are currently pinned to version 1.0.2 because of how asyncio support is patched in. Additionally, versions >2 do not include the compiled grpc headers, which are used by `pubsub_emulator.py`....
1. create a queue with a disk location 2. write a message to the queue 3. call `get` to make the message in-flight - **don't** ack or nack the message...
We are still seeing some instances of this message in the ContextualServicesSender Dataflow job in production ([example](https://console.cloud.google.com/logs/query;query=insertId%3D%228216686316375255748:267276:0:565727%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222021-07-15_09_54_23-16490022926510501865%22%20logName%3D%22projects%2Fmoz-fx-data-beam-prod-11f7%2Flogs%2Fdataflow.googleapis.com%252Fworker%22%20resource.labels.step_id%3D%2528%22WriteErrorOutput%2FBigQueryIO.Write%2FStreamingInserts%2FStreamingWriteTables%2FStreamingWrite%2FBatchedStreamingWrite.ViaBundleFinalization%2FParMultiDo%2528BatchAndInsertElements%2529%22%2529%20timestamp%20%3E%3D%20%222021-07-15T16:54:24.046Z%22%20severity%3E%3DDEFAULT;timeRange=2021-07-16T15:32:11.213Z%2F2021-07-16T15:32:11.213Z;cursorTimestamp=2021-07-16T15:32:11.212Z?authuser=0&project=moz-fx-data-beam-prod-11f7)): ``` "*~*~*~ Channel ManagedChannelImpl{logId=59, target=bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~* Make sure to...