arcade
arcade copied to clipboard
dotnet
remove hardcoded darc version from arcade-services
- [X] This issue is blocking This is blocking darc updates from making it to repositories
- [ ] This issue is causing unreasonable pain
We need to remove the hack added in https://github.com/dotnet/arcade-services/pull/1905. It looks like all issues related to https://github.com/dotnet/arcade/issues/9471 were closed, even after opening an RCA and giving details on what the next steps should be, so this work got lost, but this hack is unsustainable, and prevents us from deploying some recent fixes to darc to repos.
Some ideas on possible paths are here: https://dev.azure.com/dnceng/internal/_workitems/edit/2764
Going to try the approach where we make darc a self contained app so that it includes the needed runtime. Depending on how big the package becomes we'll see if that's an acceptable solution.
the janky version of arcade that we have in arcade-services makes producing and testing the package somewhat annoying. Now that we moved arcade-services to .net 6 I'm going to see if as part of this I can get us back in a supported arcade so that I can publish and test out the package as with the rest of our infra.
Trying out an initial set of changes to update arcade and use the regular publishing mechanism which will then let me test changes to darc more easily: https://dev.azure.com/dnceng/internal/_build/results?buildId=1878297&view=results
Attempt to bring the repo to a supported arcade is having some issues with sbom generation and I've asked @epananth to take a look. I'll continue working on just making darc self contained in the meantime.
I was able to get the repo in a supported arcade version in https://github.com/dotnet/arcade-services/pull/1951, but just came to the realization that it looks like it's not possible to have a global tool be a self contained application.
I'm not sure what our alternatives are here. The best I can come up with is for the darc-init script to install the runtime before attempting to install darc.
We could just enable roll-forward=Major for darc, then it doesn't really matter what runtime it runs on, as long as it is > what we built against.
wouldn't we need to downgrade darc to be 3.1 instead of 6.0 again for this? I can try it out and hope we don't have to spread 3.1 in a lot more places.
Yea, it would need to build against 3.1, but thats doable with a 6.0+ sdk.
I'm either doing something horribly wrong, or this approach is pretty messy.
By making darc build against 3.1, we need darclib* to build against 3.1, which means making some of the maestro projects to have to build against 3.1, which means all of maestro has to build against 3.1, which means we have to revert some of the changes that came with the upgrade to .net 6.
we shouldn't need to change that many things down the stack, only darc and things it depends on can be 3.1, or netcoreapp2.1. The other services and things can still be 6.0, and just reference things that are on 3.1
Here is the full list of projects that will need to be changed to a 3.1 target framework:
- Maestro.Contracts
- Microsoft.DotNet.Darc
- Microsoft.DotNet.DarcLib
- Microsoft.DotNet.DarcLib.AzDev
- Microsoft.DotNet.Internal.Logging
- Microsoft.DotNet.MaestroClient
- Microsoft.DotNet.Services.Utility
I might have given up one layer too early when I saw this was spreading to maestro projects but the list isn't too bad. There seems to only be one feature we were using in darc that we'll need to find an alternative for.
Error CS1061 'IEnumerable<Channel>' does not contain a definition for 'DistinctBy' and no accessible extension method 'DistinctBy' accepting a first argument of type 'IEnumerable<Channel>' could be found (are you missing a using directive or an assembly reference?)
Funnily enough, this DistinctBy that we can't use any longer when we move back to 3.1 for darc was added recently by @adiaaida in https://github.com/dotnet/arcade-services/pull/1922 and is one of the bug fixes that we want to get deployed to production. I'll find an alternative, but it's still funny.
but like.... why are we still dealing with 6.0 not being installed on the build agents? Can't we just fix that issue instead, rather than downgrading the version used to build darc?
It seems weird for the 3.1 release branches to need a globally installed 6.0 sdk in the build agents to succeed. It's definitely one way to solve this though.
Just seemed like the right thing to do was have darc work on the minimum sdk it's supposed to support.
I'm also totally fine with discussing this issue in triage instead of treating it as FR.
@adiaaida brought up a good point about this:
- 3.1 is out of support in December
- 5.0 is already out of support
With that in mind, would it make more sense to just install a .net 6 runtime in every build image (it might even be there already, I'll run some experiments..) and keep relying on the globally installed runtimes in the machines for darc to work?
Triage: we would like to try the route of installing .net 6 on the machines that require it.
Since we lost the original builds that showcased the problem, here is a more recent test build that I attempted where I updated the version of darc to use manually: https://dev.azure.com/dnceng/internal/_build/results?buildId=1895657&view=results
Plan forward:
- [x] Be able to install .NET 6 Runtime/SDK on all VS2019 build machines
- [x] .NET 6 is not available as a workload installation option for VS2019, so we'll need to make an artifact for images that have VS2019.
- [x] Validate that .NET 6 is available for VS2022
- [x] Once staging machines have this change, test getting the latest darc against staging
- [x] After rolling out OSOB, revert the hardcoded change so that we can start using the latest darc again
Well, some good news is that if we point the publish task to an image with VS2022, .NET 6 appears to already be installed with VS2022, so that's one type of image that we won't have to retrofit with .NET 6: https://dnceng.visualstudio.com/internal/_build/results?buildId=1930562&view=results
@riarenas's perspective on our findings:
From my perspective, if we're going to say "if you want to use darc, use the vs2022 images", we need to take into account:
- Servicing branches need to keep working with whatever solution we come up with. If we're going to change the eng/common templates to use vs2022, we need to coordinate with Matt Mitchell because arcade doesn't flow automatically for all the release branches in product repos.
- A deployment of arcade-services shouldn't break any of the common infrastructure, so any changes need to have flowed before we re-enable the latest versions of darc.
- There's guidance for repos that might call darc directly, and would need to move to a different pool.
PR for .NET 6 artifact: https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/24952
PR for artifact merged! Need to validate that it works when the changes roll out to staging
Ran a branch of Arcade that points to a VS2019 image on staging with a recent version of darc that uses .NET 6 and validated that it is able to publish: https://dev.azure.com/dnceng/internal/_build/results?buildId=1951792&view=results
PR to revert the hardcoding: https://github.com/dotnet/arcade-services/pull/1984
Should not merge until the OSOB changes roll out to production.
OSOB rolled out. Ricardo validated that the change in prod works, so the Arcade Services revert change has been merged!
This was a two-part fix. The first part was rolled out with OSOB this week, next week, the second part will go out with Arcade Services (doing the thing this issue is asking for).
Huzzah! It works: https://dev.azure.com/dnceng/internal/_build/results?buildId=1980010&view=logs&j=226748d0-f812-5437-d3f0-2dd291f5666e&t=bad11196-972e-5d03-45a8-9db526506073
Closing