arcade-services icon indicating copy to clipboard operation
arcade-services copied to clipboard

Maestro sometimes doesn't add darc feeds to nuget.config

Open dkurepa opened this issue 1 year ago • 3 comments

This is something I've been noticing for a while, and also something our customers reported recently: https://teams.microsoft.com/l/message/19:[email protected]/1731647680149?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&parentMessageId=1731647680149&teamName=.NET%20Core%20Eng%20Services%20Partners&channelName=First%20Responders&createdTime=1731647680149.

Basically, when we're flowing stabilized packages through Maestro, sometimes the temporary darc feed containing the package is not added to nuget.config, even tho the build has assets in it. This causes PR failures and needs manual intervention to be resolved

dkurepa avatar Nov 15 '24 09:11 dkurepa

This is because a new build of the same sha did not get published (typically because it was on a different branch). When two builds have the same sha and same version numbers for a given asset, maestro chooses the newest one. It has no easy way to disambiguate.

mmitche avatar Nov 20 '24 19:11 mmitche

We need to understand the root cause, do a spike if needed and propose + implement the fix.

michalpavelka avatar Jan 31 '25 10:01 michalpavelka

@michalpavelka The issue is pretty well understood: https://github.com/dotnet/arcade-services/issues/4153#issuecomment-2489435166

The issue is that Maestro does not know which of the input builds is the right one to use, because it only has a sha+version number in the dependency info, and that same sha and version number can be produced from any number of branches. The heuristic has been to use latest. There may be some tweaks that could be used to maybe make the result better, but it is a non-trivial problem. Coherent parent dependencies make it more complex.

Usually, it's a configuration issue that occurs when we stabilize a release branch, but an internal/release/* branch is also building due to automatic mirroring, but not publishing (due to no default channel being set up). It could also occur if a dev builds the same sha as a release branch.

Anyways, it usually gets cleaned up in servicing because we settle on a Maestro flow that doesn't build the same sha twice, typically because our internal and public branches will diverge.

I think this a low priority. .NET 10 with the VMR will largely avoid the problem altogether by avoiding the vast majority (maybe all) stable dependency flow into the VMR. It may still occasionally occur for flow out of the VMR, but I expect it will be rare.

mmitche avatar Jan 31 '25 16:01 mmitche