arcade
arcade copied to clipboard
dotnet
Move release/5.0 templates to use new build machines
- [x] This issue is blocking
- [ ] This issue is causing unreasonable pain
Reported on teams:
CI builds for SDK and installer repo for 5.0.Nxx branches are now failing to publish using darc. The CI build is just for the branding updates so likely something changed out underneath us. Pipelines - Run 20220906.7 logs (azure.com)
'D:\a\1\s.dotnet\dotnet.exe' tool install microsoft.dotnet.darc --version 1.1.0-beta.22431.1 --add-source 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json' -v minimal --tool-path 'D:\a\1\s\artifacts\tmp\Debug\darc\093436de-7d8e-41c6-8e48-6ab77f77832b'
Determining projects to restore...
C:\Users\VssAdministrator\AppData\Local\Temp\pzzqkxvu.mad\restore.csproj : error NU1202: Package Microsoft.DotNet.Darc 1.1.0-beta.22431.1 is not compatible with net5.0 (.NETCoreApp,Version=v5.0) / any. Package Microsoft.DotNet.Darc 1.1.0-beta.22431.1 supports: net6.0 (.NETCoreApp,Version=v6.0) / any
Failed to restore C:\Users\VssAdministrator\AppData\Local\Temp\pzzqkxvu.mad\restore.csproj (in 1.76 sec).
The tool package could not be restored.
Tool 'microsoft.dotnet.darc' failed to install. This failure may have been caused by:
Reopened. This is still an issue as the official build for #10749 failed with the same error. We're getting reports of repos being blocked in their 5.0 branches due to this:
https://dev.azure.com/dnceng/internal/_build/results?buildId=1985657&view=logs&j=226748d0-f812-5437-d3f0-2dd291f5666e&t=bad11196-972e-5d03-45a8-9db526506073&s=0d7db6e4-c533-50c8-77c7-b1b9fc049ed6
FYI @mmitche @wtgodbe @dougbu
I'm retrying the build but I have little hope of it fixing things.
Yep definitely not a matter of getting an older build machine. This is just still broken.
@riarenas it looks like the PR you linked was merged. Can you confirm it fixed the issue?
It didn't: https://dev.azure.com/dnceng/internal/_build/results?buildId=1994663&view=logs&j=226748d0-f812-5437-d3f0-2dd291f5666e&t=bad11196-972e-5d03-45a8-9db526506073
I don't think I'll be able to finish this by tomorrow. Kicking back to the backlog.
Most 3.1.x CI builds don't use darc; the internal/release/5.0 branches (or similar for tooling) should be prioritized for a fix.
@wtgodbe hacked dotnet-aspnetcore in https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore/commit/279fd0871a89bdcdf9fda0c79bd63a7091b0d9e4?refName=refs%2Fheads%2Finternal%2Frelease%2F5.0&path=%2Feng%2Fcommon%2Fdarc-init.ps1&_a=contents to stick w/ an old version of darc. Hopefully that's not the only choice that would work…
If this can't be fixed in a couple of days, we won't be able to take an Arcade update to address the image consolidation and Svc pool name requirements. I'd rather not get further hacks in our eng/common/ code.
Been running some experiments here, and my latest find is that the build agents don't seem to know that they have .net 6 installed.
A call to dotnet --list-sdks is only showing a 5.0 sdk being installed. https://dev.azure.com/dnceng/internal/_build/results?buildId=2011541&view=logs&j=226748d0-f812-5437-d3f0-2dd291f5666e&t=bad11196-972e-5d03-45a8-9db526506073&l=14 . Going to make myself a VM with the image to play around with.
If I can't figure things out by EOD, I can prepare a workaround PR tomorrow to use the last darc that had netcoreapp3.1 support to unblock the flow of arcade for these branches so that we get the new build images being used, while we focus on the proper fix.
We, on purpose, only install the 6.0 runtime in the build machines, which helps with running darc after installing it, but in order to install darc, we'll need to also add a 6.0 sdk. I'll prepare a fix to the image tomorrow, but since it won't roll out until next week, I'll also prepare the workaround PRs that we can use to unblock in the meantime.
I was finally able to unblock both branches and trigger the subscriptions for both .net 3 and .net 5 eng channels. The 5.0 build needed a lot of attention.
Remaining work here is to install a 6.0 sdk in the build machines and to undo some of the workarounds we had to introduce to get this unblocked.
Changes to the runtime artifact are in. I will test the removal of the workarounds by targeting the staging pool provider that should have the updated image tomorrow.
The changes have been rolled out, but when trying to remove some of the workarounds, especially the ones in the validate stages of arcade, we still end up with a mix of hosts which makes it very difficult to set up which one will be picked up for using darc, and which one will be used for the rest of the build (since the build needs to run in 3.x / 5x sdk). Given how other non-arcade scenarios are unblocked, I think we're good to close this, and keep the hardcoded versions of darc in the validate sdk legs.
I was mistaken. The changes for this have not been rolled out. I don't think we should remove the workarounds in the validate stages regardless, but I'll reopen this and put it in the right tracking column until this is actually in production proper.