container-canary CI check is flaky: "unexpected commit digest"

[jameslamb]

Description

Over the last few days, I've observed the validate jobs in CI often failing like this:

failed commit on ref "layer-sha256:2609f790a90ab5307cbbb99bd5de7e5912f440c225ffbf91122d212f48a1dfcf": commit failed: unexpected commit digest sha256:c3728131ea5aa7dddea03141cea9af41b08c03c2a6a6a0fdca0b37d13341f1f5, expected sha256:2609f790a90ab5307cbbb99bd5de7e5912f440c225ffbf91122d212f48a1dfcf: failed precondition

(recent nightly run)

This is always resolved with 1 or 2 re-runs. There's something flaky about those jobs.

Benefits of this work

Fixing this would reduce the manual effort needed to merge PRs and keep nightly workflows running successfully.

Acceptance Criteria

Nightlies succeed for 2 consecutive days with no failures in the validate jobs.

Approach

N/A

Notes

N/A

[jameslamb]

This is still happening occasionally: https://github.com/rapidsai/docker/actions/runs/21002291088/job/60377076577?pr=838#step:7:71