scons MSVC: detection fixes

This PR fixes issues surrounding MSVC detection and user specification of the vswhere executable via the construction variable VSWHERE.

This PR is similar to the MSVC detection changes originally proposed in #4409. In hindsight, it was a mistake in #4409 to move beyond the initial detection changes and attempt to incorporate preview versions.

It is my intention to abandon #4409 as it is unlikely to be adopted in any reasonable time frame due to the quantity of uncommitted local changes and the number of updates to the main branch while the branch was under development.

For purposes of exposition, the changes have been partitioned into two logical categories: detection and vswhere.

Changes:

detection:
- For msvc version specifications without an 'Exp' suffix, an express installation is used when no other edition is detected for the msvc version.
  
  Similarly, an express installation of the IDE binary is used when no other IDE edition is detected.
- VS2015 Express (14.0Exp) does not support the sdk version argument. VS2015 Express does not support the store argument for target architectures other than x86.
  
  Script argument validation now takes into account these restrictions.
- VS2015 BuildTools (14.0) does not support the sdk version argument and does not support the store argument.
  
  Script argument validation now takes into account these restrictions.
- The Windows SDK for Windows 7 and .NET Framework 4" (SDK 7.1) populates the registry keys in a manner in which the msvc detection would report that VS2010 (10.0) is installed when only the SDK was installed.
  
  The installed files are intended to be used via the sdk batch file setenv.cmd. The installed msvc batch files will fail. The msvc detection logic now ignores SDK-only VS2010 installations.
  
  Similar protection is implemented for the sdk-only installs that populate the installation folder and registry keys for VS2008 (9.0), if necessary.
- For VS2005 (8.0) to VS2015 (14.0), vsvarsall.bat is employed to dispatch to a dependent batch file when configuring the msvc environment.
  
  Previously, only the existence of the compiler executable was verified. In certain installations, the dependent batch file (e.g., vcvars64.bat) may not exist while the compiler executable does exist resulting in build failures. The existence of vcvarsall.bat, the dependent batch file, and the compiler executable are now validated.
- MSVC configuration data specific to versions VS2005 (8.0) to VS2008 (9.0) was added as the dependent batch files have different names than the batch file names used for VS2010 (10.0) and later.
  
  The VS2008 (9.0) Visual C++ For Python installation is handled as a special case as the dependent batch files are: (a) not used and (b) in different locations.
- When VS2008 (9.0) Visual C++ For Python is installed using the ALLUSERS=1 option (i.e., msiexec /i VCForPython27.msi ALLUSERS=1), the registry keys are written to HKEY_LOCAL_MACHINE rather than HKEY_CURRENT_USER.
  
  An entry was added to query the Visual C++ For Python keys in HKLM following the HKCU query, if necessary.
- For VS2008, a full development edition (e.g., Professional) is now selected before a Visual C++ For Python edition.
  
  Prior to this change, Visual C++ For Python was selected before a full development edition when both editions are installed.
- The registry detection of VS2015 (14.0), and earlier, is now cached at runtime and is only evaluated once for each msvc version.
- The detection order and ranking of msvc instances for VS 2017 and later is now deterministic.
- Visual Studio 2022 v143 BuildTools now supports msvc toolset versions from 14.30 to 14.4X.
  
  Fixes Issue #4543.
vswhere:
- The vswhere executable is frozen upon initial detection.
  
  Specifying a different vswhere executable via the construction variable VSWHERE after the initial detection now results in an exception.
  
  Multiple bugs in the implementation of specifying a vswhere executable via the construction variable VSWHERE have been fixed.
  
  Previously, when a user specified vswhere executable detects new msvc installations after the initial detection, the internal msvc installation cache, default msvc version, and msvs installation cache based on the initial detection are invalid. For example, when no vswhere executable is found for the initial detection and then later an environment is constructed with a user specified vswhere executable that detects new msvc installations.
- The vswhere detection of VS2017 (14.1), and later, is now cached at runtime and is only evaluated once using a single vswhere invocation for all msvc versions.
  
  Previously, the vswhere executable was invoked for each supported msvc version.
- The vswhere executable locations for the WinGet and Scoop package managers were added to the default vswhere executable search list after the Chocolatey installation location.

Edits:

2024/05/27: VS2022 v143 BuildTool support for msvc toolset versions from 14.30 to 14.4X.

Contributor Checklist:

[x] I have created a new test or updated the unit tests to cover the new/changed functionality.
[x] I have updated CHANGES.txt (and read the README.rst)
[x] I have updated the appropriate documentation

May 21 '24 13:05 jcbrill

It may be helpful to review the following documentation changes before reviewing the code changes:

SCons\Tool\MSCommon\README.rst
SCons\Tool\msvc.xml

The following source code files contain the bulk of the code changes:

SCons\Tool\MSCommon\vc.py
SCons\Tool\MSCommon\MSVC\Config.py
SCons\Tool\MSCommon\MSVC\Kind.py
SCons\Tool\MSCommon\MSVC\ScriptArguments.py
SCons\Tool\MSCommon\vs.py

Source file modifications:

Summary of source file modifications by category and color:

Category	Files	Description
${\tt{\normalsize{\color{Maroon}\text{source code}}}}$	2	Major source code modifications
${\tt{\normalsize{\color{GoldenRod}\text{source code}}}}$	3	Moderate source code modifications
${\tt{\normalsize{\color{Green}\text{source code}}}}$	7	Minor source code modifications
${\tt{\normalsize{\color{RoyalBlue}\text{documentation}}}}$	4	Documentation modifications
${\tt{\normalsize{\color{gray}\text{test suite}}}}$	19	Test suite modifications

Summary of source file modifications by category and reason:

Reason	Source	Docs	Tests	Description
${\tt{\normalsize{\text{detect, vswhere}}}}$	2	3	2	Detection and VSWHERE modifications
${\tt{\normalsize{\text{detect}}}}$	6	1	5	Detection modifications
${\tt{\normalsize{\text{vswhere}}}}$	3		10	VSWHERE modifications
${\tt{\normalsize{\text{internal}}}}$	1		1	Internal modifications
${\tt{\normalsize{\text{whitespace}}}}$			1	Whitespace modifications

All source code modifications by category and reason:

Edits:

2024/05/27: Added additional source files due to changes to support 14.4X msvc toolsets.

May 21 '24 13:05 jcbrill

Future work:

Modifications to support v143 toolsets 14.4X.YYYYY
Support for preview versions

Depending on the manner in which support for preview versions is implemented, it may be possible to eliminate the "kind" detection as implemented in this PR.

May 21 '24 13:05 jcbrill

Hey - this looks very comprehensive work! Wanted to note that with a dark theme (which I didn't pick, by the way, assorted defaults get me there for GitHub web pages), the table above entitled "Summary of source file modifications by category and reason:" ends up with an unreadable (black on black) Reason column. In the email notification I see it is setting that to black: ${\tt{\normalsize{\color{black}\text{detect, vswhere}}}}$ . Any possibility to tweak that?

May 21 '24 13:05 mwichmann

In the email notification I see it is setting that to black: ${\tt{\normalsize{\color{black}\text{detect, vswhere}}}}$. Any possibility to tweak that?

Removed the color assignment (${\tt{\normalsize{\text{detect, vswhere}}}}$).

Is there a difference? Having trouble reproducing.

Edit: should be fixed now.

May 21 '24 14:05 jcbrill

Is there a difference? Having trouble reproducing.

Edit: should be fixed now.

Yes, it's fine. It was only a minor nit - thanks!

May 22 '24 13:05 mwichmann

Yes, it's fine. It was only a minor nit - thanks!

I appreciate the feedback.

For something unrelated a couple of weeks ago, a script was written that was color coding SCons output from a template which would use black unless overridden. Thanks to your feedback, I learned that simply using black as a default color is a bad idea and need to use no color assignment at all for the default.

I'm guessing that abusing the gfm latex math mode for color coding is a bad idea in the long run. However, in some cases it can be effective.

May 22 '24 14:05 jcbrill

Yeah a bit off-topic, but "dark themes" have problems with this, as there's often some color assumption somewhere that doesn't work, I'm not really a fan, but sometimes it's too much trouble on websites to change to a light theme because sites keep losing the preference and changing back. Even the SCons documentation gives me this problem - when I build the API docs locally they start out white, and then as I follow links, turn black, I have no clue how that happens. Theme-aware websites are tricky despite that the frameworks all understand about it, but so many people started demanding dark themes everybody had to add those. In theory this shouldn't apply to console output text, but guess what - all the terminal emulators have dark themes too. Even the still-not-default Windows Terminal app, that can really host any "console" program, seems to default cmd and powershell to dark themes.

May 22 '24 14:05 mwichmann

Does this PR, which I haven't had a chance to actually review, deal with the versioning problem we anticipated, that is now actual? 16.10 has dropped, and with current code, it's giving me some problems. Still waiting for the test run to finish in a slow VM.

May 25 '24 21:05 mwichmann

Yeah, a bunch of not-found type of things in the failing tests,

Failed the following 7 tests:
        SCons\Tool\MSCommon\MSVC\ScriptArgumentsTests.py
        SCons\Tool\MSCommon\vcTests.py
        test\MSVC\msvc_cache_force_defaults.py
        test\MSVC\MSVC_SDK_VERSION.py
        test\MSVC\MSVC_SPECTRE_LIBS.py
        test\MSVC\MSVC_TOOLSET_VERSION.py
        test\MSVC\MSVC_UWP_APP.py

May 25 '24 22:05 mwichmann

Does this PR, which I haven't had a chance to actually review, deal with the versioning problem we anticipated, that is now actual?

Unfortunately, it does not. That was listed as future work above.

In my own work, it caused a lookup failure (python error). I would have expected something similar in SCons. I fixed it in my own work so I pretty much know an approach that should work to address part of the problem in SCons. However, there are assumptions in the SCons implementation that are going to be problematic.

I was hoping that there would be enough time to get this PR approved before having to address the toolset version issue. The best laid plans of mice and men often go awry.

I will start investigating tomorrow. Installing updates as I write this

May 26 '24 02:05 jcbrill

Couple of example test outputs... of course this is not as comprehensive as the debug logs, which I didn't capture in the initial run.

ERROR: test_msvc_script_args (__main__.ScriptArgumentsTests.test_msvc_script_args)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArgumentsTests.py", line 578, in test_msvc_script_args
    self.run_msvc_script_args()
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArgumentsTests.py", line 310, in run_msvc_script_args
    earlier_toolset_versions = [toolset_def for toolset_def in toolset_versions if toolset_def.msvc_vernum != version_def.msvc_vernum]

   ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'msvc_vernum'

======================================================================
ERROR: test_msvc_script_args_none (__main__.ScriptArgumentsTests.test_msvc_script_args_none)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArgumentsTests.py", line 573, in test_msvc_script_args_none
    self.run_msvc_script_args_none()
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArgumentsTests.py", line 292, in run_msvc_script_args_none
    _ = func(env, version_def.msvc_version, vc_dir, '')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArguments.py", line 929, in msvc_script_arguments
    toolset = _toolset_version(toolset_version)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\MSVC\ScriptArguments.py", line 190, in _toolset_version
    vs_def = Config.MSVC_VERSION_INTERNAL[verstr]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: '14.4'

May 26 '24 12:05 mwichmann

We need a new issue.

Resolving the lookup error is fairly straightforward by effectively adding an indirection.

Immediately, the issues are:

There is an implicit assumption that the toolset prefix is equivalent to the msvc version. That is no longer true: 14.40.33807 => 14.4 is not a valid MSVC_VERSION which is the indexing failure. Most of the issues are in the ScriptArguments validation.
vc_version and msvc_version no longer mean the same thing. The vc_version 14.4 maps to build tools v143 which at present is only available in msvc_version 14.3. For lack of a better name, a build series data structure is introduced (e.g., 14.4, 14.3, ...) and the build tools definition was changed to a list of build series (e.g., v143: [14.4, 14.3]).

May 26 '24 13:05 jcbrill

I am attempting to resolve the issue in this PR.

May 26 '24 13:05 jcbrill

I'll create a new issue if we don't have one - I remember kicking the potential problem around a few times before. It seems the regexes fail here, as you say - it is finding cl.exe for the version, but then

  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\vc.py", line 1734, in msvc_query_version_toolset
    raise MSVCArgumentError(msg)
SCons.Tool.MSCommon.MSVC.Exceptions.MSVCArgumentError: Unsupported msvc version '14.40.33806'

and then, eventually, if it hasn't died by then:

KeyError: '14.4':

which is directly the translation of 14.40 to 14.4, as you say.

May 26 '24 13:05 mwichmann

We started a discussion, but I never got around to populating the content.

FYI: unrelated news, the fix for the issue where the vcvarsall batch files would fail when vcpkg is installed and msvs is installed in path with a close parenthesis (e.g., "c:\program files (x86)") appears in the latest update.

May 26 '24 13:05 jcbrill

@mwichmann Where can I find what is installed in the new windows Full Test Suite?

The test is still in progress and at least one test (SCons\Tool\MSCommon\vcTests.py) has failed.

A toolset query succeeded when the expectation was that it should fail. It would be helpful to know what versions of Visual Studio and which toolsets are installed.

This was tested against quite a few VS configurations. However, there were some changes made today after all pcs and vms were upgraded to the latest toolsets (14.4x). I'm wondering if the test environment only has 14.3x toolsets.

The code changes have been more or less straightforward but the test suite changes have been annoyingly tedious.

May 28 '24 01:05 jcbrill

@mwichmann Where can I find what is installed in the new windows Full Test Suite?

The test is still in progress and at least one test (SCons\Tool\MSCommon\vcTests.py) has failed.

The problems with that build, and some details about it, are described in #4539. That list does not have anything for the new Visual Studio version, which is not provisioned into the runner.

May 28 '24 12:05 mwichmann

The problems with that build, and some details about it, are described in https://github.com/SCons/scons/issues/4539. That list does not have anything for the new Visual Studio version, which is not provisioned into the runner.

I was looking for all of the actual toolset versions installed.

After thinking about the test that failed, I have convinced myself that older toolsets may also be installed (e.g., 14.29, 14.16) which would cause the failed test to fail. I'm about to push a fix soon.

Is the MSVC cache enabled? Surprised by the total run time.

May 28 '24 12:05 jcbrill

For my build that showed the MSVC errors pasted above, the set of installed things is very controlled - just v14.40-17.10.

For the github builder - you're right, I didn't get the cache setup there, teach me to hack out anything quickly. Now I have to figure out how to set environment vars in a github runner....

May 28 '24 13:05 mwichmann

... and to send things too quickly. The provisioning information for that puppy can be found here:

https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md

May 28 '24 13:05 mwichmann

... and to send things too quickly. The provisioning information for that puppy can be found here:

https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md

I was looking at that last night. From my first reading, it didn't look like additional toolsets are installed. In a local vm, with only vs2022 and older toolsets installed, the same test failed.

The test fix in question passed prior to the full test suite being canceled.

May 28 '24 13:05 jcbrill

I killed the pending build here and created #4545

May 28 '24 13:05 mwichmann

Sorry for messing with things... Bill is also working on a PR which reduces tool initialization by adding those pesky DefaultEnvironment(tools=[]) calls, that's in #4544. We'll get to a stable state soon and not get in your way...

May 28 '24 13:05 mwichmann

Sorry for messing with things...

No worries. It actually worked out fine. It was past the test of interest. Making a few tweaks locally anyway so no harm done.

When you have really nothing better to do, this branch should work for VS2022 w/14.40 now.

May 28 '24 13:05 jcbrill

The overall build in #4545 went much faster, and after some more fiddles, is merged.

May 28 '24 16:05 mwichmann

Oh, there was a question on toolsets on the github image, this is probably relevant: https://github.com/actions/runner-images/issues/9701

May 28 '24 16:05 mwichmann

Sorry for messing with things...

No worries. It actually worked out fine. It was past the test of interest. Making a few tweaks locally anyway so no harm done.

When you have really nothing better to do, this branch should work for VS2022 w/14.40 now.

I will try to test, having a lot of other things taking my time.

May 29 '24 15:05 mwichmann

@mwichmann off-topic for the documentation sheriff: There are typographical and grammatical issues with the Reproducible Builds section of README.rst.

Jun 02 '24 15:06 jcbrill

@mwichmann off-topic for the documentation sheriff: There are typographical and grammatical issues with the Reproducible Builds section of README.rst.

Okay, will have a look.

Jun 02 '24 15:06 mwichmann

@mwichmann off-topic for the documentation sheriff: There are typographical and grammatical issues with the Reproducible Builds section of README.rst.

Okay, will have a look.

Wow... that (and the referenced template) are pretty messy....

Jun 02 '24 15:06 mwichmann