data-infra
data-infra copied to clipboard
Published
20 hours ago •
cal-itp
cal-itp
Update to GTFS schedule validator v7.1 (#3763)
Description
Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.
- Upgrades GTFS validator to v7.1: adds rules etc.
- Updates the build & push GitHub workflow to push to development image when on a branch to facilitate testing of PodOperator
- Moves to uv-based workflow
Resolves #3763
Reviewer notes:
- May need to drop 93c20570d82c28d1b52598f8dc28d6ef56a37410 and 602b943c796856163c97f687d9ede632c74861b9 if testing locally (sets effective date to 12/1) -- if merging after that date, this should be bumped back
- Refactor on top of #4532? Defer to @ohrite and @erikamov as to whether this should be merged in current form
Type of change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation
How has this been tested?
Include commands/logs/screenshots as relevant.
If making changes to dbt models, please run the command poetry run dbt run -s CHANGED_MODEL and poetry run dbt test -s CHANGED_MODEL, then include the output in this section of the PR.
- Runs in local Composer for 10/25/25 (picked as random date)
- Can see validations in
laurie_staging.stg_gtfs_schedule__validation_noticesbut they don't filter into the fct model because the job times out in local Composer (I think it's just an issue with local k8s resources?) so a validation outcomes file isn't generated, which is required for notices to get into the fct model
Post-merge follow-ups
Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.
- [ ] No action required
- [ ] Actions required (specified below)
If any reviewer looks at this -- setting a past cutover date for testing purposes, will bump
Terraform plan in iac/cal-itp-data-infra-staging/airflow/us
Plan: 1 to add, 21 to change, 3 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
!~ update in-place
- destroy
Terraform will perform the following actions:
# google_storage_bucket_object.calitp-staging-composer["dags/airtable_loader_v2/generate_gtfs_download_configs.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "MijWlA==" -> (known after apply)
!~ detect_md5hash = "s632w01yc8uo408y4VdAyw==" -> "different hash"
!~ generation = 1763690417635115 -> (known after apply)
id = "calitp-staging-composer-dags/airtable_loader_v2/generate_gtfs_download_configs.py"
!~ md5hash = "s632w01yc8uo408y4VdAyw==" -> (known after apply)
name = "dags/airtable_loader_v2/generate_gtfs_download_configs.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["dags/dags.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "K+/MxQ==" -> (known after apply)
!~ detect_md5hash = "dZgOhCDXnrL508gF1WVo3g==" -> "different hash"
!~ generation = 1763083967978599 -> (known after apply)
id = "calitp-staging-composer-dags/dags.py"
!~ md5hash = "dZgOhCDXnrL508gF1WVo3g==" -> (known after apply)
name = "dags/dags.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["dags/download_gtfs_schedule_v2/download_schedule_feeds.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "8qLecA==" -> (known after apply)
!~ detect_md5hash = "iGapm0xJ3U0wowUUkId1eQ==" -> "different hash"
!~ generation = 1763690417218909 -> (known after apply)
id = "calitp-staging-composer-dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
!~ md5hash = "iGapm0xJ3U0wowUUkId1eQ==" -> (known after apply)
name = "dags/download_gtfs_schedule_v2/download_schedule_feeds.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "g2TBuw==" -> (known after apply)
!~ detect_md5hash = "PSes9rK7j0FP6JcNRQhPLg==" -> "different hash"
!~ generation = 1763690417804807 -> (known after apply)
id = "calitp-staging-composer-dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
!~ md5hash = "PSes9rK7j0FP6JcNRQhPLg==" -> (known after apply)
name = "dags/sync_ntd_data_xlsx/scrape_ntd_xlsx_urls.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "WAOTIw==" -> (known after apply)
!~ detect_md5hash = "Q2KZHpa145QxEt0qrMVrvQ==" -> "different hash"
!~ generation = 1751929677147947 -> (known after apply)
id = "calitp-staging-composer-dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
!~ md5hash = "Q2KZHpa145QxEt0qrMVrvQ==" -> (known after apply)
name = "dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/__init__.py"] will be destroyed
# (because key ["plugins/calitp_data_infra/__init__.py"] is not in for_each map)
- resource "google_storage_bucket_object" "calitp-staging-composer" {
- bucket = "calitp-staging-composer" -> null
- content_type = "text/plain; charset=utf-8" -> null
- crc32c = "AAAAAA==" -> null
- detect_md5hash = "1B2M2Y8AsgTpgAmY7PhCfg==" -> null
- event_based_hold = false -> null
- generation = 1763690417199832 -> null
- id = "calitp-staging-composer-plugins/calitp_data_infra/__init__.py" -> null
- md5hash = "1B2M2Y8AsgTpgAmY7PhCfg==" -> null
- md5hexhash = "d41d8cd98f00b204e9800998ecf8427e" -> null
- media_link = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2F__init__.py?generation=1763690417199832&alt=media" -> null
- metadata = {} -> null
- name = "plugins/calitp_data_infra/__init__.py" -> null
- output_name = "plugins/calitp_data_infra/__init__.py" -> null
- self_link = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2F__init__.py" -> null
- source = "../../../../airflow/plugins/calitp_data_infra/__init__.py" -> null
- storage_class = "STANDARD" -> null
- temporary_hold = false -> null
# (6 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/auth.py"] will be destroyed
# (because key ["plugins/calitp_data_infra/auth.py"] is not in for_each map)
- resource "google_storage_bucket_object" "calitp-staging-composer" {
- bucket = "calitp-staging-composer" -> null
- content_type = "text/plain; charset=utf-8" -> null
- crc32c = "6lsUtA==" -> null
- detect_md5hash = "+/KTbwc3sd3B4wBkY+HoUw==" -> null
- event_based_hold = false -> null
- generation = 1763690417207979 -> null
- id = "calitp-staging-composer-plugins/calitp_data_infra/auth.py" -> null
- md5hash = "+/KTbwc3sd3B4wBkY+HoUw==" -> null
- md5hexhash = "fbf2936f0737b1ddc1e3006463e1e853" -> null
- media_link = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fauth.py?generation=1763690417207979&alt=media" -> null
- metadata = {} -> null
- name = "plugins/calitp_data_infra/auth.py" -> null
- output_name = "plugins/calitp_data_infra/auth.py" -> null
- self_link = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fauth.py" -> null
- source = "../../../../airflow/plugins/calitp_data_infra/auth.py" -> null
- storage_class = "STANDARD" -> null
- temporary_hold = false -> null
# (6 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/calitp_data_infra/storage.py"] will be destroyed
# (because key ["plugins/calitp_data_infra/storage.py"] is not in for_each map)
- resource "google_storage_bucket_object" "calitp-staging-composer" {
- bucket = "calitp-staging-composer" -> null
- content_type = "text/plain; charset=utf-8" -> null
- crc32c = "b87VYA==" -> null
- detect_md5hash = "sq1Q+wmsL8o0RKJLFUFC7g==" -> null
- event_based_hold = false -> null
- generation = 1763690417395966 -> null
- id = "calitp-staging-composer-plugins/calitp_data_infra/storage.py" -> null
- md5hash = "sq1Q+wmsL8o0RKJLFUFC7g==" -> null
- md5hexhash = "b2ad50fb09ac2fca3444a24b154142ee" -> null
- media_link = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fstorage.py?generation=1763690417395966&alt=media" -> null
- metadata = {} -> null
- name = "plugins/calitp_data_infra/storage.py" -> null
- output_name = "plugins/calitp_data_infra/storage.py" -> null
- self_link = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Fcalitp_data_infra%2Fstorage.py" -> null
- source = "../../../../airflow/plugins/calitp_data_infra/storage.py" -> null
- storage_class = "STANDARD" -> null
- temporary_hold = false -> null
# (6 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/hooks/kuba_hook.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "pIf6jA==" -> (known after apply)
!~ detect_md5hash = "M9n0Cr7dL9+4asfxnMHgjQ==" -> "different hash"
!~ generation = 1763690418428084 -> (known after apply)
id = "calitp-staging-composer-plugins/hooks/kuba_hook.py"
!~ md5hash = "M9n0Cr7dL9+4asfxnMHgjQ==" -> (known after apply)
name = "plugins/hooks/kuba_hook.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/hooks/soda_hook.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "Wxj+aQ==" -> (known after apply)
!~ detect_md5hash = "CDAoj9pONq2nsD5BT43vEg==" -> "different hash"
!~ generation = 1763690418284709 -> (known after apply)
id = "calitp-staging-composer-plugins/hooks/soda_hook.py"
!~ md5hash = "CDAoj9pONq2nsD5BT43vEg==" -> (known after apply)
name = "plugins/hooks/soda_hook.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/hooks/transitland_hook.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "xolZ/w==" -> (known after apply)
!~ detect_md5hash = "lMSV7OyTWTBE5ar8Ush1oA==" -> "different hash"
!~ generation = 1763690417210931 -> (known after apply)
id = "calitp-staging-composer-plugins/hooks/transitland_hook.py"
!~ md5hash = "lMSV7OyTWTBE5ar8Ush1oA==" -> (known after apply)
name = "plugins/hooks/transitland_hook.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/aggregator_to_gcs_operator.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "h26Utg==" -> (known after apply)
!~ detect_md5hash = "+2mzXgNPW36mWf5Z3ZZBag==" -> "different hash"
!~ generation = 1763690417144134 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/aggregator_to_gcs_operator.py"
!~ md5hash = "+2mzXgNPW36mWf5Z3ZZBag==" -> (known after apply)
name = "plugins/operators/aggregator_to_gcs_operator.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/blackcat_to_gcs_operator.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "s7BJ7Q==" -> (known after apply)
!~ detect_md5hash = "husvfrVLOwWESpUczaAT3w==" -> "different hash"
!~ generation = 1763690417861255 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/blackcat_to_gcs_operator.py"
!~ md5hash = "husvfrVLOwWESpUczaAT3w==" -> (known after apply)
name = "plugins/operators/blackcat_to_gcs_operator.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/dbt_manifest_to_dictionary_operator.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "4MyK1Q==" -> (known after apply)
!~ detect_md5hash = "FIHwYcjOm5NPB+nPb7enyg==" -> "different hash"
!~ generation = 1763690416913428 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/dbt_manifest_to_dictionary_operator.py"
!~ md5hash = "FIHwYcjOm5NPB+nPb7enyg==" -> (known after apply)
name = "plugins/operators/dbt_manifest_to_dictionary_operator.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/dbt_manifest_to_metadata_operator.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "UDX9og==" -> (known after apply)
!~ detect_md5hash = "F4n9FAx9ExF1Zl3J7NaMWQ==" -> "different hash"
!~ generation = 1763690417995295 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/dbt_manifest_to_metadata_operator.py"
!~ md5hash = "F4n9FAx9ExF1Zl3J7NaMWQ==" -> (known after apply)
name = "plugins/operators/dbt_manifest_to_metadata_operator.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/gtfs_csv_to_jsonl_hourly.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "pCzW9g==" -> (known after apply)
!~ detect_md5hash = "n34Iv1u6y/wZeOnqO2WDEA==" -> "different hash"
!~ generation = 1763690417647780 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/gtfs_csv_to_jsonl_hourly.py"
!~ md5hash = "n34Iv1u6y/wZeOnqO2WDEA==" -> (known after apply)
name = "plugins/operators/gtfs_csv_to_jsonl_hourly.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/littlepay_raw_sync_feed_v3.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "TDDjLg==" -> (known after apply)
!~ detect_md5hash = "zXyVjenO8Z0Xcx4u85EVYQ==" -> "different hash"
!~ generation = 1763690417211071 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/littlepay_raw_sync_feed_v3.py"
!~ md5hash = "zXyVjenO8Z0Xcx4u85EVYQ==" -> (known after apply)
name = "plugins/operators/littlepay_raw_sync_feed_v3.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/pod_operator.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "H81Llg==" -> (known after apply)
!~ detect_md5hash = "6vO0LHE3p5d/cOQ71Ghv8g==" -> "different hash"
!~ generation = 1763690417200759 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/pod_operator.py"
!~ md5hash = "6vO0LHE3p5d/cOQ71Ghv8g==" -> (known after apply)
name = "plugins/operators/pod_operator.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/scrape_ntd_xlsx.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "800p7w==" -> (known after apply)
!~ detect_md5hash = "+sUY5347tlkwmkjx/59Ytg==" -> "different hash"
!~ generation = 1763690417819438 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/scrape_ntd_xlsx.py"
!~ md5hash = "+sUY5347tlkwmkjx/59Ytg==" -> (known after apply)
name = "plugins/operators/scrape_ntd_xlsx.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/operators/scrape_state_geoportal.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "HSTs0g==" -> (known after apply)
!~ detect_md5hash = "kroDAzyyod9g32UxYWccew==" -> "different hash"
!~ generation = 1763690418239800 -> (known after apply)
id = "calitp-staging-composer-plugins/operators/scrape_state_geoportal.py"
!~ md5hash = "kroDAzyyod9g32UxYWccew==" -> (known after apply)
name = "plugins/operators/scrape_state_geoportal.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/scripts/gtfs_rt_parser.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "XjBzGA==" -> (known after apply)
!~ detect_md5hash = "BfvE2ctBSfWnSYBpIv2hOg==" -> "different hash"
!~ generation = 1763690417837883 -> (known after apply)
id = "calitp-staging-composer-plugins/scripts/gtfs_rt_parser.py"
!~ md5hash = "BfvE2ctBSfWnSYBpIv2hOg==" -> (known after apply)
name = "plugins/scripts/gtfs_rt_parser.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer["plugins/utils.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~ crc32c = "2wseZw==" -> (known after apply)
!~ detect_md5hash = "L7u5luMr8c8OXJVHR7nzEg==" -> "different hash"
!~ generation = 1763690417826500 -> (known after apply)
id = "calitp-staging-composer-plugins/utils.py"
!~ md5hash = "L7u5luMr8c8OXJVHR7nzEg==" -> (known after apply)
name = "plugins/utils.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "FZrGzA==" -> (known after apply)
!~ detect_md5hash = "xTWjkGqqzCqWNDhJxp6R/w==" -> "different hash"
!~ generation = 1749663120395357 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"
!~ md5hash = "xTWjkGqqzCqWNDhJxp6R/w==" -> (known after apply)
name = "data/warehouse/models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_quality/_mart_gtfs_quality.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "Rs71zQ==" -> (known after apply)
!~ detect_md5hash = "rnq255v0R46m5jQzuY/z0g==" -> "different hash"
!~ generation = 1749663113676778 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/mart/gtfs_quality/_mart_gtfs_quality.yml"
!~ md5hash = "rnq255v0R46m5jQzuY/z0g==" -> (known after apply)
name = "data/warehouse/models/mart/gtfs_quality/_mart_gtfs_quality.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"] will be created
+ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
+ bucket = "calitp-staging-composer"
+ content = (sensitive value)
+ content_type = (known after apply)
+ crc32c = (known after apply)
+ detect_md5hash = "different hash"
+ generation = (known after apply)
+ id = (known after apply)
+ kms_key_name = (known after apply)
+ md5hash = (known after apply)
+ md5hexhash = (known after apply)
+ media_link = (known after apply)
+ name = "data/warehouse/seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"
+ output_name = (known after apply)
+ self_link = (known after apply)
+ source = "../../../../warehouse/seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"
+ storage_class = (known after apply)
}
Plan: 1 to add, 21 to change, 3 to destroy.
:memo: Plan generated in Plan Terraform for Warehouse and DAG changes #1078
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Terraform plan in iac/cal-itp-data-infra/airflow/us
Plan: 1 to add, 4 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
!~ update in-place
Terraform will perform the following actions:
# google_storage_bucket_object.calitp-composer["dags/dags.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer" {
!~ crc32c = "K+/MxQ==" -> (known after apply)
!~ detect_md5hash = "dZgOhCDXnrL508gF1WVo3g==" -> "different hash"
!~ generation = 1763057268028226 -> (known after apply)
id = "calitp-composer-dags/dags.py"
!~ md5hash = "dZgOhCDXnrL508gF1WVo3g==" -> (known after apply)
name = "dags/dags.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer["dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer" {
!~ crc32c = "WAOTIw==" -> (known after apply)
!~ detect_md5hash = "Q2KZHpa145QxEt0qrMVrvQ==" -> "different hash"
!~ generation = 1751416672802943 -> (known after apply)
id = "calitp-composer-dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
!~ md5hash = "Q2KZHpa145QxEt0qrMVrvQ==" -> (known after apply)
name = "dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "FZrGzA==" -> (known after apply)
!~ detect_md5hash = "xTWjkGqqzCqWNDhJxp6R/w==" -> "different hash"
!~ generation = 1751416661426004 -> (known after apply)
id = "calitp-composer-data/warehouse/models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"
!~ md5hash = "xTWjkGqqzCqWNDhJxp6R/w==" -> (known after apply)
name = "data/warehouse/models/intermediate/gtfs_quality/int_gtfs_quality__schedule_validator_rule_details_unioned.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/mart/gtfs_quality/_mart_gtfs_quality.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "Rs71zQ==" -> (known after apply)
!~ detect_md5hash = "rnq255v0R46m5jQzuY/z0g==" -> "different hash"
!~ generation = 1751416667981834 -> (known after apply)
id = "calitp-composer-data/warehouse/models/mart/gtfs_quality/_mart_gtfs_quality.yml"
!~ md5hash = "rnq255v0R46m5jQzuY/z0g==" -> (known after apply)
name = "data/warehouse/models/mart/gtfs_quality/_mart_gtfs_quality.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"] will be created
+ resource "google_storage_bucket_object" "calitp-composer-dags" {
+ bucket = "calitp-composer"
+ content = (sensitive value)
+ content_type = (known after apply)
+ crc32c = (known after apply)
+ detect_md5hash = "different hash"
+ generation = (known after apply)
+ id = (known after apply)
+ kms_key_name = (known after apply)
+ md5hash = (known after apply)
+ md5hexhash = (known after apply)
+ media_link = (known after apply)
+ name = "data/warehouse/seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"
+ output_name = (known after apply)
+ self_link = (known after apply)
+ source = "../../../../warehouse/seeds/gtfs_schedule_validator_rule_details_v7_1_0.csv"
+ storage_class = (known after apply)
}
Plan: 1 to add, 4 to change, 0 to destroy.
:memo: Plan generated in Plan Terraform for Warehouse and DAG changes #1078
Terraform plan in iac/cal-itp-data-infra-staging/composer/us
Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
Terraform will perform the following actions:
# google_composer_environment.calitp-staging-composer will be updated in-place
!~ resource "google_composer_environment" "calitp-staging-composer" {
id = "projects/cal-itp-data-infra-staging/locations/us-west2/environments/calitp-staging-composer"
name = "calitp-staging-composer"
# (5 unchanged attributes hidden)
!~ config {
# (8 unchanged attributes hidden)
!~ software_config {
!~ pypi_packages = {
+ "calitp-data-infra" = "==2025.6.5"
+ "pydantic" = ">=1.9,<2.0"
# (11 unchanged elements hidden)
}
# (6 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# (8 unchanged blocks hidden)
}
# (1 unchanged block hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
:memo: Plan generated in Plan Terraform for Warehouse and DAG changes #1078
Terraform plan in iac/cal-itp-data-infra/composer/us
No changes. Your infrastructure matches the configuration.
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
:memo: Plan generated in Plan Terraform for Warehouse and DAG changes #1078
Warehouse report 📦
DAG
Legend (in order of precedence)
| Resource type | Indicator | Resolution |
|---|---|---|
| Large table-materialized model | Orange | Make the model incremental |
| Large model without partitioning or clustering | Orange | Add partitioning and/or clustering |
| View with more than one child | Yellow | Materialize as a table or incremental |
| Incremental | Light green | |
| Table | Green | |
| View | White |
There is an issue with the bucket path, it's being constructed with 2025-10-25T02:06:06.716612Z format instead of 2025-10-25T02:06:06.716612+00:00 format