telemetry-airflow
telemetry-airflow copied to clipboard
mozilla
Airflow configuration for Telemetry
The following PR introduces a job for evaluating whether our recorded search terms are changing in ways that might invalidate assumptions on which we've built our sanitization model: https://github.com/mozilla/docker-etl/pull/73 THIS...
The PR would resolve the issue where the syncs are completed quickly after being started by the FivetranOperator. More details here: https://github.com/fivetran/airflow-provider-fivetran/issues/60 The Fivetran connector v1.1.2 will help us avoid...
Bumps [numpy](https://github.com/numpy/numpy) from 1.13.3 to 1.22.0. Release notes Sourced from numpy's releases. v1.22.0 NumPy 1.22.0 Release Notes NumPy 1.22.0 is a big release featuring the work of 153 contributors spread...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Bumps [numpy](https://github.com/numpy/numpy) from 1.13.3 to 1.22.0. Release notes Sourced from numpy's releases. v1.22.0 NumPy 1.22.0 Release Notes NumPy 1.22.0 is a big release featuring the work of 153 contributors spread...
Bumps [pyspark](https://github.com/apache/spark) from 2.3.2 to 3.1.3. Commits d1f8a50 Preparing Spark release v3.1.3-rc4 7540421 Preparing development version 3.1.4-SNAPSHOT b8c0799 Preparing Spark release v3.1.3-rc3 0a7eda3 [SPARK-38075][SQL][3.1] Fix hasNext in HiveScriptTransformationExec's pro... 91db9a3...
Next step is to modify the import job container to subscribe to the topic and verify msgs prior to importing. This should be done in dev/stage first with separate subscriptions....
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.6.1 to 3.15.0. Release notes Sourced from protobuf's releases. Protocol Buffers v3.15.0 Protocol Compiler Optional fields for proto3 are enabled by default, and no longer require the...
`dataproc_init.sh` runs pip 20.3.1 with the new resolver and the requirements that it's trying to install don't gel well with one another causing the resolver to take infinite time to...
See https://github.com/mozilla/telemetry-airflow/pull/982#discussion_r423706798 We have multiple places now where we set a group of options together in order to ensure bigquery_etl_query replaces a whole table rather than just a partition. It's...
There's interest in developing some stronger safeguards around pipelines that feed publicly released data. Pipeline code generally does not live in this repository, but this repo does serve as a...