telemetry-airflow

telemetry-airflow copied to clipboard

Jobs run with GKEOperator need `get_logs=False`, otherwise job is likely to fail unless constantly logging to standard out

2

I noticed this while working on adding the missioncontrol-etl job (#840), but apparently this happened with the probe scraper as well. tl;dr: a job must print something to standard out...

wlach

There is now a `tmp` dataset for temporary tables used in intermediate operations. We should utilize that dataset for the export.

fbertsch

That function is used in one place, so it should live in it's own script. That script should be called by `bin/run`.

fbertsch

The `make run` command does not work properly because the mysql database takes too long to respond. The wait script does not check whether the database is accepting requests, and...

acmiyaguchi

Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.6.1 to 3.18.3. Release notes Sourced from protobuf's releases. Protocol Buffers v3.18.3 C++ Reduce memory consumption of MessageSet parsing This release addresses a Security Advisory for C++...

dependabot[bot]

Add Glam DEV legacy telemetry dag to start testing hot probes list

1

This creates a new Dev DAG for Glam, which will be used to test the modified bigquery-etl code (not yet landed) that only computes probes that have been recently created...

edugfilho

DENG-2928: Merged publish_bqetl_static dag into bqetl_artifact_deployment dag

1

Decision has been made to use `metadata.yaml` files instead of `description.txt` for static tables. As a result, the static tables will rely on `publish metadata` job to publish metadata. See...

wwyc

Replace SubDAGs with TaskGroups (bug 1866141)

2

## [Bug 1866141](https://bugzilla.mozilla.org/show_bug.cgi?id=1866141): Replace Airflow SubDAGs with TaskGroups [SubDAGs](https://airflow.apache.org/docs/apache-airflow/2.6.3/core-concepts/dags.html#subdags) were deprecated in Airflow v2.0, and [it appears they'll be removed entirely in v3.0](https://github.com/apache/airflow/blob/fcb91f47c9833a956e67745c2dadc7a533282b78/airflow/operators/subdag.py#L101). [TaskGroups](https://airflow.apache.org/docs/apache-airflow/2.6.3/core-concepts/dags.html#taskgroups) are Airflow's [recommended](https://airflow.apache.org/docs/apache-airflow/2.6.3/core-concepts/dags.html#taskgroups-vs-subdags) alternative to SubDAGs,...

sean-rose

feat: create two new dags for the Firefox-CI ETL

3

## Description ## Related Tickets & Documents * [Bug 1904928](https://bugzilla.mozilla.org/show_bug.cgi?id=1904928) (tracking) * [DENG-4195](https://mozilla-hub.atlassian.net/browse/DENG-4195) (secrets) [DENG-4195]: https://mozilla-hub.atlassian.net/browse/DENG-4195?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

ahal

chore(bqetl_artifact_deployment): unschedule DAG

1

## Description Requires https://github.com/mozilla-services/cloudops-infra/pull/5736. We should then be running bqetl artifact deploys after every merge to main and every new schemas deploy. Note that on days where there are no...

whd