Mirror dependencies for release branches

[phlax]

There has been some offline discussion about a longer term solution for the issue that we were hit by twice last week involving hash changes for an upstream tarball where the upstream did not functionally change.

This occurred for 2 separate reasons last week

In one case the upstream is (afaiaa) an internal google repo that for some reason (possibly timestamp leakage) does not guarantee hash consistency in produced tarballs even when there is no change. For this reason the tarball has historically been mirrored for Envoys consumption to provide that consistency. In this case (iiuc) there was some problem with the mirroring process and the hash did change when the tarball was remirrored

There was a separate issue where the name of an upstream repo was changed. Altho the old tarball redirected to the new repo the path inside the tarball had changed and the hash no longer matched.

There is a related problem in that github does not guarantee hash consistency anyway, and has a couple of times in the past recompressed tarballs across github breaking many builds across the interwebs

Altho these issues are rare they pose a problem for anyone trying to build from historical branches, as we generally only patch the currently supported ones and anything expecting the originally hashed tarballs will now break.

One solution that has been proposed is to effectively mirror all of the deps for a release branch when its cut

There are few hurdles/issues with this approach:

• setup/maintenance of CI to keep mirror correct/up-to-date
• as the dep files tend to be large, at least when initially populating the mirror a lot of CI time/resources may be required
• to be of any use the mirror will have to be kept working for much longer than our (1 year) support branch policy period
• some logic will need to be figured out wrt choosing between the authoritative or mirrored files on main and release branches
• likewise some logic will need to be figured out for testing changes on the release branches, assuming the change involves a dependency that is not currently mirrored

there are various strategies we could employ to mitigate or resolve these issues but the setup/maintenance does require some work

relatedly we have considered a similar strategy in the past due to transient flakiness downloading from github itself, altho that was addressed in a different way

[phlax]

cc @kyessenov @ggreenway @alyssawilk

[phlax]

see here for some prior art https://github.com/grpc/grpc/blob/master/bazel/update_mirror.sh

[phlax]

issue for client caching is here https://github.com/envoyproxy/envoy/issues/36340

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.