Doc Ritezel

@dependabot recreate

As discussed 6/20 w/ @lottspot: 1. Merge https://github.com/cal-itp/data-infra/pull/3908 2. Monitor production DAG runs for errors 3. Create and resolve any bugs that arise from DAG changes in the current production...

# Rollout plan for DAGs First batch - [x] airtable_loader_v2 (no dependencies, low impact) - [x] create_external_tables (no dependencies, low impact) - [x] copy_prod_archiver_configs_to_test Second batch - [x] transform_warehouse (runs...

Paired with @erikamov today to discuss how to control increasing costs over time. Erika will write up a brief summary for review with the group and @evansiroky.

As discussed at planning, @erikamov will meet with @evansiroky and analysts to figure out the lowest-hanging fruit. After discussion, we want to discuss GTFS-RT validation outputs.

One way we can introspect on BigQuery compute usage is by adding labels to individual tables/views: https://docs.getdbt.com/reference/resource-configs/bigquery-configs

Still to be done as of 3/4: documentation for tables and columns, backfill tests

Pydantic was updated to 2, we need to validate each DAG in order to merge this

This is a low-code solution for GTFS-RT archiving, using: * [Cloud Scheduler](https://console.cloud.google/cloudscheduler?csesidx=1127815958&project=cal-itp-data-infra-staging) * [Eventarc](https://console.cloud.google/eventarc?csesidx=1127815958&project=cal-itp-data-infra-staging) * [Workflows](https://console.cloud.google/workflows?csesidx=1127815958&project=cal-itp-data-infra-staging) There are two workflows: 1. Gather URLs and secret settings from BigQuery and push...

I’ve had some thoughts on how to upgrade to Composer 2 latest over the last few months. The biggest issues are that the Airflow DAGs aren’t covered by automated tests,...