dotnet
Consider moving to the official "centralized package version" support
NuGet has had an official way to centrally manage package versions for some time now (https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-packages, https://docs.microsoft.com/en-us/dotnet/core/compatibility/sdk/5.0/directory-packages-props-imported-by-default, https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-package-versions).
It does this via a Directory.Packages.props file and has better integration with the IDE and other tooling.
We should look at migrating arcade dependent repositories to using the official tooling shipped alongside the produce where possible, rather than relying on custom solutions. If the official tooling is missing minor support, then we should ensure the relevant tracking issues are filed to enable the scenarios or that our custom needs are built on top where feasible.
cc @mmitche @ViktorHofer @markwilkie This refers to the conventions described in https://github.com/dotnet/arcade/blob/e7ede87875f41a9b3df898ae08da5ebc96e24f56/Documentation/ArcadeConventions.md#dependent-packages-version.
Changes could potentially impact darc as well.
This sounds like goodness (obviously). Any downsides folks are aware of?
To be transparent, we're very short of room to do stuff right now that's not security - so any help would be super appreciated.
There is one downside, which is that we can't get rid of Versions.props or all versions in the Versions.props file. There are a number of versions that are used within the product which are not NuGet package references (e.g. versions used to point to blob locations for files that the SDK might download). You probably could move those into the Directory.Packages.props file, but you're still left with a solution where Versions.props exists (it's used for product versioning).
Can you use properties in Directory.Packages.props? We already use central package version management in arcade-services, we just reference the properties in Versions.props https://github.com/dotnet/arcade-services/blob/main/eng/Packages.props
I'm supportive, I just don't want to end up with infrastructure which is more complicated than what we already have.
Its just an automatically imported props file and one that the IDE tooling knows how to reference and handle.
You can put anything in it and can even put PackageVersion items in other files as well (I don't know how good the IDE support is if you do the latter, however).
There is no reason we can't just import Versions.props inside the Directory.Packages.props, then use the version properties in the file.
I would prefer that the package versions actually exist in the Directory.Packages.props file though, for the IDE support that may be coming.
There is also a bit of work to remove the versions from all the individual csproj files. And if there is any existing difference in versions between various projects, resolving those (those it's probably wise to resolve those anyway)
Note that this also means we either need to force people to suppress NU1507, or maestro will need to add a mapping for every single package that it's adding to that list too and clean them up.
To work around it, I meanwhile did the following for dotnet/runtime:
<packageSourceMapping>
<packageSource key="dotnet-public">
<package pattern="*" />
</packageSource>
<packageSource key="dotnet-tools">
<package pattern="*" />
</packageSource>
<packageSource key="dotnet-eng">
<package pattern="*" />
</packageSource>
<packageSource key="dotnet7">
<package pattern="*" />
</packageSource>
<packageSource key="dotnet7-transport">
<package pattern="*" />
</packageSource>
<packageSource key="richnav">
<package pattern="*" />
</packageSource>
</packageSourceMapping>
That's a good starting point anyway to use package sources. You can then step by step add patterns for each feed.
I'm currently looking into using NuGet's Central Package Management in dotnet/runtime and I'm now at a point where I understand all the changes that we would need from Darc/Maestro so that it works well. Would it make sense to set up a quick meeting next week and discuss the requirements? Who should be part of that discussion? @mmitche @ChadNedzlek and @alexperovich?
@ViktorHofer be careful, if you check that it, it will break when it's time to ship, because the stabilization feeds won't work. It's better to just ignore the warning until maestro supports it.
I'd also like to make sure expectations are understood here. We currently don't have plans to support this in maestro during the .NET 7 timeframe.
Oh! The the official build will likely break as well, because all the packages in ".packages" will find your Directory.* files and import them, and then bar publishing will fail. We needed to do https://github.com/dotnet/arcade-services/pull/1892 to block packages/artifacts from crawling back up and finding our files. But if you didn't do it in the root, that wouldn't cause a problem.
(Also they had to be added with "git add -f", because those directories are .gitignored)
See my comment in https://github.com/dotnet/arcade-services/pull/1892#issuecomment-1126495688.
I'd also like to make sure expectations are understood here. We currently don't have plans to support this in maestro during the .NET 7 timeframe.
Understood. I'm willing to help and send a PR to the component that is responsible for updating the Versions.props file. But before that I would like to discuss our requirements and if those make sense.
Yeah let's get together and discuss this. If it works, I'd prefer to simple import Versions.props in the package management file.
Also, you should suppress NU1507, the package source mapping will work with stable versioning.
Stopped doing that, because I ran into some blockers that we need newer versions of to get updates to this preview feature.
Mostly this is just "wait until they fix their blocking issue, then reapply my changes". Hopefully it shouldn't be too hard to reapply... hopefully?
K. Yeah, I'm trying to figure out how to tell if they've fixed their blocking issue. :-) Trying to figure out which changes are in which nuget is annoyingly difficult.
To work around it, I meanwhile did the following for dotnet/runtime:
<packageSourceMapping> <packageSource key="dotnet-public"> <package pattern="*" /> </packageSource> <packageSource key="dotnet-tools"> <package pattern="*" /> </packageSource> <packageSource key="dotnet-eng"> <package pattern="*" /> </packageSource> <packageSource key="dotnet7"> <package pattern="*" /> </packageSource> <packageSource key="dotnet7-transport"> <package pattern="*" /> </packageSource> <packageSource key="richnav"> <package pattern="*" /> </packageSource> </packageSourceMapping>That's a good starting point anyway to use package sources. You can then step by step add patterns for each feed.
Wait I could have done that instead of suppressing the warnings from it for when I use multiple feeds in a repo?
That will not work, because maestro will add new feeds, and it won't add the packageSource, causing the warning/error to reappear. The warning needs to be disabled, otherwise it's a timebomb waiting that will make the release process a nightmare.
@ChadNedzlek will you check in with the folks that worked on this stuff to see if the bug that was blocking us from doing the work is still blocking this?
I believe the issue that was the blocking bug was https://github.com/NuGet/Home/issues/10311. It was resolved in July 2022. I think, based on the tagging in that repo, that should mean that it's in anything higher than NuGet version 6.4.0.31? But according to https://learn.microsoft.com/en-us/nuget/release-notes/nuget-6.4, 6.5 hasn't been released yet, so if we are using anything less than that, it's... sort of arbitrary if it's included or not?
It's probably been long enough that it's probably there. But depending on the fourth part of the version number isn't something that's easy to check/assess. Especially since the dependency is rather indirect (when we are building, I think we are getting it through the SDK of dotnet, I think?)
The result of that bug is that if you used transitive pinning (sort of the whole point of trying to enable it) "metadata" doesn't flow right, including private assets, which we make heavy use of in arcade.
It appears that the latest version of dotnet has a nuget version that is high enough for us to try this again. Going to drop this in Operations. We should think of doing it next time we have to upgrade packages
Arcade is now using central package management. https://github.com/dotnet/arcade-services/issues/3002 tracks support on the maestro side for the managed feeds and package source mapping.