sdk
sdk copied to clipboard
dart-lang
Build reproducibility issue on Windows caused issues for 3.4.4 release
These builds were triggered on the same commits: https://ci.chromium.org/ui/p/dart-internal/builders/ci/dart-sdk-win-arm64-stable/42/overview https://ci.chromium.org/ui/p/dart-internal/builders/ci/dart-sdk-win-arm64-stable/43/overview
They created these CIPD packages: https://chrome-infra-packages.appspot.com/p/dart/dart-sdk/windows-arm64/+/BJQj5t8FiV7ro7FDJSGRGDZf3p05YsCdagql2lSq2-AC https://chrome-infra-packages.appspot.com/p/dart/dart-sdk/windows-arm64/+/syZDaczXKFTFVfvT13KA4jL6sSHuhY-nHSPJqWLTwSIC
The packages have slight differences (handful of bytes) in these binaries:
bin/dart.exe
bin/dartaotruntime.exe
bin/utils/gen_snapshot.exe
bin/utils/wasm-opt.exe
This caused the CIPD packages to have different hashes, which in turn caused an inconsistency because there are now two packages with the same git_version tag but different content.
The differences are in the PE/COFF timestamps and a GUID in the RSDS reference to the PDB file.
It looks like ldd-link offers some flags to handle the COFF timestamp, but we are still using the MSVC linker even when we compile with Clang. (https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html)
@athomas looks like we have always had this issue and it is not a recent regressions, I would like to understand the criticality of this problem (i.e is it something we have to fix?)
It's probably a P2 (depending on the effort needed to fix it), impact is:
- Introduces risk in the release process: re-running Windows builds doesn't work if they flake at the wrong moment (after the SDK CIPD package is uploaded and tagged but before the BCID attestation is added). These broken CIPDs need to be untagged manually to allow the releases to go out. This happens rarely.
- Caches for RBE builds on Windows will be less effective. This probably happens all the time, but just degrades build speeds (especially on Windows).