Supporting local SDK deployment in global.json

[jaredpar]

trafficstars

This proposal extends global.json such that it supports locally deployed instances of the .NET SDK. This will significantly reduce friction our local deployment has with developer tools like Visual Studio, VS Code and make our own developer story much simpler.

[xoofx]

Thanks for writing this proposal! Looks like it's covering our use case for using a custom dotnet SDK with Unity + MSBuild.

[baronfel]

Just a note that if we take this we should update the official schema for global.json files that's over in SchemaStore to declare the new properties (and link to docs for them).

[jaredpar]

Are you proposing a change to the failure case? For instance "I need 9.0.100-preview.2 exactly and it must be in .dotnet but the user hasn't run the script to make that happen".

This proposal is narrowly focused on changing the host resolver algorithm only. This means it should be effectively a transparent change to the rest of the tooling stack. If the correct SDK was not found it would emit an error just as it does today. Maybe the error text updates a bit to list the locations considered (not sure if that is surfaced today). It would not attempt anything like auto-acquiring the correct SDK.

[agocke]

Confirming: this is an SDK-only feature, correct? That is, the behavior of the app host or of dotnet blah.dll would not be affected? That appears to be true, as written. However, I would change the "host resolver" to simply "SDK resolver" to remove confusion.

[sharwell]

We (Roslyn team) are desperately in need of this fix.

[CyrusNajmabadi]

FYI, this is probably my biggest pain point being a dev on hte roslyn team. Constant breakages here are pretty frustrating :)

[baronfel]

One thing to consider: how does this impact CI/CD setups like GitHub Actions' actions/setup-dotnet? I'd hope that those actions could learn to check for this new field and install the on-demand SDK to that location?

[RussKie]

Something has been bothering me about the proposal, and I think I finally figured out what that was. The developer experience, or the lack of, when the required SDK is absent.

In my opinion there are two gaps in the current functionality:

1. The ability to specify where the developer wants the tooling (such as VS) to look for the SDK - and this proposal is tacking this.
2. The ability to inform a contributor what to do when the SDK is missing.

If we consider the typical use case - a contributor clones a repo, which declares in global.json what version of the SDK is required and where it is supposed to be located, double clicks on a solution file, and.... nothing works. VS tells the contributor that it can't resolves types, find assemblies and build the solution. This is a very poor developer experience. Building the command line is somewhat better - it'd tell that the required SDK is nowhere to be found, lists the available SDKs, but the end results is still poor - the contributor may not know how to install the required SDK.

A way to improve this experience could be the ability to optionally specify a custom message that'd be shown if the required SDK is absent. E.g., something like this:

{
    "sdk": {
        "additionalPaths": [ ".dotnet" ],
        "installMessage": "Please run restore.cmd to install the required tooling.",
    }
}

[gab]

Something has been bothering me about the proposal, and I think I finally figured out what that was. The developer experience, or the lack of, when the required SDK is absent. [...]

If I can chime in for a minute (Hi guys!), in the case of my team (and we're awaiting this functionality with baited breath), we'd likely be including the SDK itself in the repo and have the global.json point to it through a relative path.

You still raise a good point though. Sometimes you might not want to bloat your source control history with all those successive SDKs, and leaving the user dead in the water with no idea what to do isn't great.

And you're reminding me of one of the ways I was researching to solve this: it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source - or a local NuGet repo containing a copy of an official package. As far as I can tell most of the infrastructure is in place for this (due to support for custom SDKs in NuGet format, albeit barely documented), except that Microsoft doesn't provide official .NET SDKs through NuGet. I'm not sure why, given how portable .NET is these days. I'm perfectly ok with requiring an older .NET version for bootstrapping.

So, I realize that this is tangential to the current proposal, but I feel like automated official .NET SDK download is a use case that isn't supported well either. Perhaps this proposal can help open the way for it, or at least try not to interfere with this alternate use case.

[akoeplinger]

it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source

This is essentially how most dotnet GitHub repos work, the build.sh/.ps1 scripts for commandline builds use a helper script called tools.sh/tools.ps1 which looks into global.json and installs the corresponding SDK into a local .dotnet folder if the system dotnet isn't sufficient. Having an official way to do that from the CLI would be nice, but I think this is orthogonal to this proposal.

[gab]

it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source

This is essentially how most dotnet GitHub repos work, the build.sh/.ps1 scripts for commandline builds use a helper script called tools.sh/tools.ps1 which looks into global.json and installs the corresponding SDK into a local .dotnet folder if the system dotnet isn't sufficient. Having an official way to do that from the CLI would be nice, but I think this is orthogonal to this proposal.

I've already said that much (re: orthogonality), but I think that the desire of having a unified experience between command line builds and opening a solution in Visual Studio is the same, so there might be a common base to both problems.

[mmitche]

it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source

This is essentially how most dotnet GitHub repos work, the build.sh/.ps1 scripts for commandline builds use a helper script called tools.sh/tools.ps1 which looks into global.json and installs the corresponding SDK into a local .dotnet folder if the system dotnet isn't sufficient. Having an official way to do that from the CLI would be nice, but I think this is orthogonal to this proposal.

I like the idea of having a standardized acquisition process, though it's not entirely straightforward because the default sources for the SDK are not necessarily sufficient to obtain the SDK (e..g you might need non-official locations).

[mmitche]

it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source

This is essentially how most dotnet GitHub repos work, the build.sh/.ps1 scripts for commandline builds use a helper script called tools.sh/tools.ps1 which looks into global.json and installs the corresponding SDK into a local .dotnet folder if the system dotnet isn't sufficient. Having an official way to do that from the CLI would be nice, but I think this is orthogonal to this proposal.

I like the idea of having a standardized acquisition process, though it's not entirely straightforward because the default sources for the SDK are not necessarily sufficient to obtain the SDK (e..g you might need non-official locations).

[RussKie]

it'd be real nice to be able to just specify a .NET SDK version and have some global.json parameter to order NuGet to fetch it from an official source

This is essentially how most dotnet GitHub repos work, the build.sh/.ps1 scripts for commandline builds use a helper script called tools.sh/tools.ps1 which looks into global.json and installs the corresponding SDK into a local .dotnet folder if the system dotnet isn't sufficient. Having an official way to do that from the CLI would be nice, but I think this is orthogonal to this proposal.

I like the idea of having a standardized acquisition process, though it's not entirely straightforward because the default sources for the SDK are not necessarily sufficient to obtain the SDK (e..g you might need non-official locations).

I avoided mentioning this, as I have already floated that idea and it was rejected by @richlander citing "security concerns". Although, that rejection was countered by @akoeplinger in this comment. I appreciate the security angle, but I'm with @akoeplinger on this, the security issue exists regardless but the rejection of this option makes the developer experience substandard.

[sharwell]

Windows Update KB5033741 just broke developers working on Roslyn.sln, again (by silently changing the installed SDK version). We are in ongoing desperate need of this fix.

[sharwell]

Windows Update KB5033741 just broke developers working on Roslyn.sln, again (by silently changing the installed SDK version). We are in ongoing desperate need of this fix.

[agocke]

FYI this is not a fix. This would be a new feature. No matter what we implement, it will not be backported to .NET 8.

[agocke]

FYI this is not a fix. This would be a new feature. No matter what we implement, it will not be backported to .NET 8.

[agocke]

Thanks for all the feedback everyone!

Given that the muxer is the riskiest component in the SDK, I'd like to propose a narrower change that hopefully meets the same needs. I propose we add exactly one property of the following form:

{
    "sdk": {
        "paths": [ "$host$", ".dotnet" ]
    }
}

When the paths variable is null or missing the SDK resolution algorithm proceeds as it does today.

When present, paths effectively overrides the search paths entirely. The paths will be searched in order to find an SDK with the requested version. Resolution will halt whenever one is found.

There is exactly one special variable: "$host$". It resolves to the path of the currently running muxer (dotnet.exe) executable. It is equivalent to the current SDK search directory. Any number of paths may be added into the list. If they are absolute paths, they will be resolved according to standard path resolution. If they are relative paths, they will be resolved relative to the global.json file.

And that's it. No other customization is provided. Would this address the most pressing scenarios?

[agocke]

Thanks for all the feedback everyone!

Given that the muxer is the riskiest component in the SDK, I'd like to propose a narrower change that hopefully meets the same needs. I propose we add exactly one property of the following form:

{
    "sdk": {
        "paths": [ "$host$", ".dotnet" ]
    }
}

When the paths variable is null or missing the SDK resolution algorithm proceeds as it does today.

When present, paths effectively overrides the search paths entirely. The paths will be searched in order to find an SDK with the requested version. Resolution will halt whenever one is found.

There is exactly one special variable: "$host$". It resolves to the path of the currently running muxer (dotnet.exe) executable. It is equivalent to the current SDK search directory. Any number of paths may be added into the list. If they are absolute paths, they will be resolved according to standard path resolution. If they are relative paths, they will be resolved relative to the global.json file.

And that's it. No other customization is provided. Would this address the most pressing scenarios?

[vitek-karas]

I think it's a good solution for now, and we'll see if it works for everybody or it needs more tweaking. I only question the support for absolute paths. I don't see it being very useful, plus it feels like potential security trouble (but I don't have a specific case in mind).

[vitek-karas]

I think it's a good solution for now, and we'll see if it works for everybody or it needs more tweaking. I only question the support for absolute paths. I don't see it being very useful, plus it feels like potential security trouble (but I don't have a specific case in mind).

[agocke]

I only question the support for absolute paths. I don't see it being very useful, plus it feels like potential security trouble (but I don't have a specific case in mind).

Agreed on the questionable usefulness. I hope it doesn't pose any security risks, as I think you could emulate the behavior by using a relative path with a very large number of .. directories.

[agocke]

I only question the support for absolute paths. I don't see it being very useful, plus it feels like potential security trouble (but I don't have a specific case in mind).

Agreed on the questionable usefulness. I hope it doesn't pose any security risks, as I think you could emulate the behavior by using a relative path with a very large number of .. directories.

[RussKie]

What about the developer experience part? The current "the SDK not found" experience is really bad.

[RussKie]

What about the developer experience part? The current "the SDK not found" experience is really bad.

[agocke]

No opinion on the proposed customizable error message. @elinor-fung , @vitek-karas any thoughts there?

[agocke]

No opinion on the proposed customizable error message. @elinor-fung , @vitek-karas any thoughts there?

[vitek-karas]

I think it would make sense to be able to add some custom message to the error. Ideally, when we fail to find the SDK, we should:

• Print out the fact that we could not find an SDK and specify which version we were looking for
• Print out the locations we looked into (basically the resolved list of paths from the global.json if it's used)
• Append the custom message if there is one

[vitek-karas]

I think it would make sense to be able to add some custom message to the error. Ideally, when we fail to find the SDK, we should:

• Print out the fact that we could not find an SDK and specify which version we were looking for
• Print out the locations we looked into (basically the resolved list of paths from the global.json if it's used)
• Append the custom message if there is one

[elinor-fung]

I think just the paths support already addresses the main pain, but a customizable error message would be reasonable. Currently, after listing the requested version, global.json file, and installed SDKs, we print a generic instruction like:

Install the [9.0.100-alpha.1.23615.4] .NET SDK or update [C:\helloworld\global.json] to match an installed SDK.

which could be replaced by any custom message.

[elinor-fung]

I think just the paths support already addresses the main pain, but a customizable error message would be reasonable. Currently, after listing the requested version, global.json file, and installed SDKs, we print a generic instruction like:

Install the [9.0.100-alpha.1.23615.4] .NET SDK or update [C:\helloworld\global.json] to match an installed SDK.

which could be replaced by any custom message.

[agocke]

OK, I think we can accept the design from https://github.com/dotnet/designs/pull/303#issuecomment-1902268838, plus a customizable error message field. I think we can also schedule this work for .NET 9.

@jaredpar Do you want to update this proposal to that effect, or should I create a new proposal?

[agocke]

OK, I think we can accept the design from https://github.com/dotnet/designs/pull/303#issuecomment-1902268838, plus a customizable error message field. I think we can also schedule this work for .NET 9.

@jaredpar Do you want to update this proposal to that effect, or should I create a new proposal?

[jaredpar]

Do you want to update this proposal to that effect, or should I create a new proposal?

I can update this proposal (or if you want feel free to push the changes).

[RussKie]

I think just the paths support already addresses the main pain, but a customizable error message would be reasonable. Currently, after listing the requested version, global.json file, and installed SDKs, we print a generic instruction like:

Install the [9.0.100-alpha.1.23615.4] .NET SDK or update [C:\helloworld\global.json] to match an installed SDK.

which could be replaced by any custom message.

Not sure about the "replace" part as this would open a new can of worms such as formatting of the string. In my mind a custom message would be an addition to the standard message.

[RussKie]

I think just the paths support already addresses the main pain, but a customizable error message would be reasonable. Currently, after listing the requested version, global.json file, and installed SDKs, we print a generic instruction like:

Install the [9.0.100-alpha.1.23615.4] .NET SDK or update [C:\helloworld\global.json] to match an installed SDK.

which could be replaced by any custom message.

Not sure about the "replace" part as this would open a new can of worms such as formatting of the string. In my mind a custom message would be an addition to the standard message.

[jaredpar]

Just want to confirm that our editors/dev tools use this entry point.

@rainersigwald can you confirm this?

[jaredpar]

Just want to confirm that our editors/dev tools use this entry point.

@rainersigwald can you confirm this?

[simonferquel]

Just asking for confirmation: if I have a custom MSBuild SDK (that extends the .net SDK) will I be able to use those paths instead of relying on nuget to resolve it ? - the spec only mentions the .net SDK, not custom ones.

[baronfel]

@simonferquel correct - this is only about automatically using a locally-installed .NET SDK. The MSBuild SDKs feature (which has an unfortunately-similar name) is entirely different.

[MartyIX]

Do I understand correctly that this would allow me to: Have globally installed .NET 8 but test with .NET 9 that would be stored in a folder (not installed)?

I always want to test my .NET MAUI app on .NET 9 but it's not practical to install it and uninstall it every time as I want to use .NET 8 as my default .NET.

[agocke]

@MartyIX you can do this today by installing .NET 9, but using global.json to specify a .NET 8 SDK version. That will cause your build to always use .NET 8. When you want to test on .NET 9 you can temporarily bump that version to .NET 9, then revert the change when you’re done.