Quarterly Maintenance: Google Cloud Composer 2025Q3/Q4

[ohrite]

Quarterly Maintenance:

Prior Work

Quarterly Maintenance: 2025Q1/Q2

Dependencies

The major dependency upgrades to be moved forward during this maintenance period are:

• Airflow (2.6 ➡️ 2.10)

Acceptance Criteria

• [ ] Describe the upgrade methodology and any specific steps involved (unless already documented in prior work).
• [ ] Document the upgrade process and any relevant changes.
• [ ] Create issues around any roadblocks encountered.
• [ ] Conduct thorough testing to verify functionality after the upgrade and patching.

Rollback Plan

1. Backfilling tests should not change baseline functionality
2. Porting Python scripts with Gusty frontmatter to the Hook/Connection/Operator style should not change baseline functionality
3. Replacing PodOperator with the Hook/Connection/Operator style will not change baseline functionality
4. Migrating from calitp_data_infra.storage to Hook/Connection/Operator style will potentially change some functionality, but test coverage should prevent regressions

Notes

Please enter any additional information that will facilitate the completion of this maintenance task. For example: Are there any constraints not mentioned above? Are there any alternatives you have considered?

[mjumbewu]

~~As production composer is currently at composer-2.9.7-airflow-2.7.3, I am starting on testing composer-2.9.7-airflow-2.9.3 in staging.~~

See issue #4333

[mjumbewu]

As we move to Airflow 2.10+ (and eventually 3+), we should be careful about deprecations. See the results of poetry run pytest (e.g. at https://github.com/cal-itp/data-infra/actions/runs/18145803545/job/51647082401#step:10:79) for example.