SCons not defining VCINSTALLDIR or VSINSTALLDIR in env['MSVS'] per documentation

[bdbaddog]

This issue was originally created at: 2015-02-13 15:22:03. This issue was reported by: dirkbaechle. dirkbaechle said at 2015-02-13 15:22:03

As reported by Andrew C. Morrow, the 'MSVS' Environment variable contents appear to be out of sync with their documentation. The SCons 2.3.4 docs state:

MSVS
    When the Microsoft Visual Studio tools are initialized, they set up this dictionary with the following keys:

And then lists several keys, including PLATFORMSDKDIR.

However the documented variables do not actually appear to get set. The string PLATFORMSDKDIR, for instance, appears nowhere in the SCons 2.3.4 sources.

dirkbaechle said at 2015-02-13 15:22:34

Added keyword

[mwichmann]

On a fully setup Visual Studio environment, it populates the MSVS dictionary only with two keys: PROJECTSUFFIX and SOLUTIONSUFFIX. Looks like quite a bit of the rest of the MS* construction variable setting differs from the manpage too.

[acmorrow]

When this gets addressed, I think it would also be handy to introduce a VSWHERE SCons variable. This could be set at environment construction to tell SCons which vswhere.exe to use (since it can be installed in different places). Alternatively, if not set, and SCons locates it itself, it should set VSWHERE, so that user build systems can also make use of VSWHERE without needing to implement their own search.

[mwichmann]

Okay, for the first part of this, why would it matter which one you use? Supplying a path that we wouldn't have looked at otherwise seems fine, although it appears if you have any product that can be scanned with vswhere, it will have installed vswhere in the known places.

[acmorrow]

I thought only VS 2017+ installed vswhere? Imagine that I have a VS 2015 only machine, and I want to use vswhere to locate the runtime redistributables. It would be nice to be able to pull down the chocolatey version, and then specify VSWHERE=... on my command line (having made a Variable decl for it in my SConstruct). On other platforms, where SCons will find vswhere, I don't need to do this (SCons would populate it for me from the expected location), but I can still read out the discovered location for vswhere from VSWHERE. So any tooling I need that wants vswhere can be made independent of its actual install location. The same code will work with VS 2017 and VS 2019. Another reason would be if the "system" version of vswhere had a bug and you could get the newer chocolatey version and override which one SCons would use to fix a bug. Overall, it just seems smart to be able to let the user have control over this as an input parameter, the same way MSVC_VERSION works. Or any other number of situations where the default search for vswhere proves to not work as advertised, this gives the user a path forward.

[bdbaddog]

vswhere can detect some "legacy" versions older than 2017. So there is value.

[acmorrow]

Or, look it as future proofing for when Microsoft in their wisdom decides to move vswhere somewhere new in a future version.

[mwichmann]

Fair enough.

As to it working on pre-2017 - as I said in discord, my system that has 2015/2017/2019 all installed doesn't have vswhere "see" anything about the 2015 edition. Updated: with the -legacy flag it does, although it presents only three bits of information about a legacy release - should be enough to work with.

[acmorrow]

Given that this feature appears to have been broken for some time, I think it is worthwhile considering what we would like the fix to look like. I'm not sure that simply re-introducing the prior MSVS dictionary is the right fix.

Let's look at each in turn:

• MSVS.VERSION: This says it is the version in use, which can be set by MSVS_VERSION, which in turn is deprecated in favor of MSVC_VERSION. If we already have MSVC_VERSION why do we need this additional form? I'd propose eliminating it.

• MSVS.VERSIONS: I'm not sure how useful it is to expose this to users. Typically, either they requested an explicit version by setting MSVC_VERSION during initialization, in which case they know what they wanted, or they didn't, in which case they are happy with latest. So, I also propose eliminating this.

• MSVS.VCINSTALLDIR: This seems useful, and should be kept in some form.

• MSVS.VSINSTALLDIR: Ditto.

• MSVS.FRAMEWORK*: I'd propose that either all of these be dropped, or, if there is real use for knowing these, that they be simplified and renamed. I'd drop FRAMEWORKVERSIONS under the same argument as MSVS.VERSIONS. I'd also stick DOTNET in front of the name of each.

• MSVS.PLATFORMSDK: This seems useful. I wonder, in fact, whether it ought to be in/out like MSVC_VERSION is, so that you could potentially point to a newer SKD installation? But on the other hand, MSSDK_DIR and MSSDK_VERSION already exist.

• MSVS.PLATFORMSDK_MODULES: No idea here. But if MSSDK_DIR and MSSDK_VERSION obsolete MSVS.PLATFORMSDK*, then I'd propose just making this a top-level MSSDK_MODULES.

Finally, I think I'd propose hoisting all of these out of the MSVS dictionary. Having these things inside a dictionary isn't great, because it makes it difficult to set them via the command line and Variables objects.

My suggestion is that we end up with top level variables as follows:

• New top-level MSVS_INSTALLDIR and MSVC_INSTALLDIR. We could optionally make that INSTALL_DIR.
• A new top-level MSSDK_MODULES to replace MSVS.PLATFORMSDK_MODULES, if that makes sense.

I'm open to suggestions on what to do with the .NET stuff.

[acmorrow]

@bdbaddog - Any thoughts on my comments above? And should we tag it for 4.0? It seems worth fixing - there was definitely a regression somewhere along the line, and exposing the installation directories is handy for things like locating the vcredist files.

[bdbaddog]

May or may not make it for 4.0.0 but tagging for now.

[bdbaddog]

Or, look it as future proofing for when Microsoft in their wisdom decides to move vswhere somewhere new in a future version.

Note VSWHERE env variable has been implemented for some time at present.

[jcbrill]

Populating the MSVS dictionary with most of the values listed in the documentation appears to have been removed prior to the release of scons-2.0.0-final.0. The function responsible for populating the dictionary was present in scons-1.2.0 and is shown below.

It appears that starting with SCons 2.0, most of the detection routines were moved into dedicated modules. Some of the dictionary elements are available now via (possibly internal) module functions.

All of the content appears to be "pure detection" via register queries and file system examination rather than the result of running the vcvars batch files. The vcvars batch files produce many of the variables of interest but are not necessarily preserved in the resulting environment. For example, when running the vcvars batch files, VCINSTALLDIR is propagated to the resulting ENV dictionary while VSINSTALLDIR is not.

Note: scons-1.2.0 appears to have supported vs versions 6.0 (and possibly earlier), 7.0 (2002), 7.1 (2003), and 8.0 (2005).

VC/VS:

• VERSION
• VERSIONS
• VCINSTALLDIR
• VSINSTALLDIR

.NET Framework:

• FRAMEWORKDIR (r'Software\Microsoft.NETFramework\InstallRoot')
• FRAMEWORKVERSIONS (filtered os.listdir(rv['FRAMEWORKDIR']))
• FRAMEWORKVERSION
• FRAMEWORKSDKDIR (r'Software\Microsoft.NETFramework\sdkInstallRoot' + ver)

Platform SDK:

• PLATFORMSDKDIR (r'Software\Microsoft\MicrosoftSDK\Directories\Install Dir')
• PLATFORMSDK_MODULES (r'Software\Microsoft\MicrosoftSDK\InstalledSDKs')

Detection functionality largely moved to independent source modules:

• VS: scons-master/SConst/Tool/MSCommon/vs.py
• VC: scons-master/SConst/Tool/MSCommon/vc.py
• .NET Framework: scons-master/SCons/Tool/MSCommon/netframework.py
• Platform SDK: scons-master/SCons/Tool/MSCommon/sdk.py

scons-1.2.0/engine/SCons/Tool/msvs.py

get_msvs_install_dirs function (lines: 1340-1521)

def get_msvs_install_dirs(version = None, vs8suite = None):
    """
    Get installed locations for various msvc-related products, like the .NET SDK
    and the Platform SDK.
    """

    if not SCons.Util.can_read_reg:
        return {}

    if not version:
        versions = get_visualstudio_versions()
        if versions:
            version = versions[0] #use highest version by default
        else:
            return {}

    version_num, suite = msvs_parse_version(version)

    K = 'Software\\Microsoft\\VisualStudio\\' + str(version_num)
    if (version_num >= 8.0):
        if vs8suite == None:
            # We've been given no guidance about which Visual Studio 8
            # suite to use, so attempt to autodetect.
            suites = get_visualstudio8_suites()
            if suites:
                vs8suite = suites[0]

        if vs8suite == 'EXPRESS':
            K = 'Software\\Microsoft\\VCExpress\\' + str(version_num)

    # vc++ install dir
    rv = {}
    if (version_num < 7.0):
        key = K + r'\Setup\Microsoft Visual C++\ProductDir'
    else:
        key = K + r'\Setup\VC\ProductDir'
    try:
        (rv['VCINSTALLDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE, key)
    except SCons.Util.RegError:
        pass

    # visual studio install dir
    if (version_num < 7.0):
        try:
            (rv['VSINSTALLDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE,
                                                             K + r'\Setup\Microsoft Visual Studio\ProductDir')
        except SCons.Util.RegError:
            pass

        if not rv.has_key('VSINSTALLDIR') or not rv['VSINSTALLDIR']:
            if rv.has_key('VCINSTALLDIR') and rv['VCINSTALLDIR']:
                rv['VSINSTALLDIR'] = os.path.dirname(rv['VCINSTALLDIR'])
            else:
                rv['VSINSTALLDIR'] = os.path.join(SCons.Platform.win32.get_program_files_dir(),'Microsoft Visual Studio')
    else:
        try:
            (rv['VSINSTALLDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE,
                                                             K + r'\Setup\VS\ProductDir')
        except SCons.Util.RegError:
            pass

    # .NET framework install dir
    try:
        (rv['FRAMEWORKDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE,
            r'Software\Microsoft\.NETFramework\InstallRoot')
    except SCons.Util.RegError:
        pass

    if rv.has_key('FRAMEWORKDIR'):
        # try and enumerate the installed versions of the .NET framework.
        contents = os.listdir(rv['FRAMEWORKDIR'])
        l = re.compile('v[0-9]+.*')
        installed_framework_versions = filter(lambda e, l=l: l.match(e), contents)

        def versrt(a,b):
            # since version numbers aren't really floats...
            aa = a[1:]
            bb = b[1:]
            aal = string.split(aa, '.')
            bbl = string.split(bb, '.')
            # sequence comparison in python is lexicographical
            # which is exactly what we want.
            # Note we sort backwards so the highest version is first.
            return cmp(bbl,aal)

        installed_framework_versions.sort(versrt)

        rv['FRAMEWORKVERSIONS'] = installed_framework_versions

        # TODO: allow a specific framework version to be set

        # Choose a default framework version based on the Visual
        # Studio version.
        DefaultFrameworkVersionMap = {
            '7.0'   : 'v1.0',
            '7.1'   : 'v1.1',
            '8.0'   : 'v2.0',
            # TODO: Does .NET 3.0 need to be worked into here somewhere?
        }
        try:
            default_framework_version = DefaultFrameworkVersionMap[version[:3]]
        except (KeyError, TypeError):
            pass
        else:
            # Look for the first installed directory in FRAMEWORKDIR that
            # begins with the framework version string that's appropriate
            # for the Visual Studio version we're using.
            for v in installed_framework_versions:
                if v[:4] == default_framework_version:
                    rv['FRAMEWORKVERSION'] = v
                    break

        # If the framework version couldn't be worked out by the previous
        # code then fall back to using the latest version of the .NET
        # framework
        if not rv.has_key('FRAMEWORKVERSION'):
            rv['FRAMEWORKVERSION'] = installed_framework_versions[0]

    # .NET framework SDK install dir
    if rv.has_key('FRAMEWORKVERSION'):
        # The .NET SDK version used must match the .NET version used,
        # so we deliberately don't fall back to other .NET framework SDK
        # versions that might be present.
        ver = rv['FRAMEWORKVERSION'][:4]
        key = r'Software\Microsoft\.NETFramework\sdkInstallRoot' + ver
        try:
            (rv['FRAMEWORKSDKDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE,
                key)
        except SCons.Util.RegError:
            pass

    # MS Platform SDK dir
    try:
        (rv['PLATFORMSDKDIR'], t) = SCons.Util.RegGetValue(SCons.Util.HKEY_LOCAL_MACHINE,
            r'Software\Microsoft\MicrosoftSDK\Directories\Install Dir')
    except SCons.Util.RegError:
        pass

    if rv.has_key('PLATFORMSDKDIR'):
        # if we have a platform SDK, try and get some info on it.
        vers = {}
        try:
            loc = r'Software\Microsoft\MicrosoftSDK\InstalledSDKs'
            k = SCons.Util.RegOpenKeyEx(SCons.Util.HKEY_LOCAL_MACHINE,loc)
            i = 0
            while 1:
                try:
                    key = SCons.Util.RegEnumKey(k,i)
                    sdk = SCons.Util.RegOpenKeyEx(k,key)
                    j = 0
                    name = ''
                    date = ''
                    version = ''
                    while 1:
                        try:
                            (vk,vv,t) = SCons.Util.RegEnumValue(sdk,j)
                            # TODO(1.5):
                            #if vk.lower() == 'keyword':
                            #    name = vv
                            #if vk.lower() == 'propagation_date':
                            #    date = vv
                            #if vk.lower() == 'version':
                            #    version = vv
                            if string.lower(vk) == 'keyword':
                                name = vv
                            if string.lower(vk) == 'propagation_date':
                                date = vv
                            if string.lower(vk) == 'version':
                                version = vv
                            j = j + 1
                        except SCons.Util.RegError:
                            break
                    if name:
                        vers[name] = (date, version)
                    i = i + 1
                except SCons.Util.RegError:
                    break
            rv['PLATFORMSDK_MODULES'] = vers
        except SCons.Util.RegError:
            pass

    return rv

generate function (lines: 1732-1790, fragment: 1766-1776)

def generate(env):
...
    try:
        version = get_default_visualstudio_version(env)
        # keep a record of some of the MSVS info so the user can use it.
        dirs = get_msvs_install_dirs(version)
        env['MSVS'].update(dirs)
    except (SCons.Util.RegError, SCons.Errors.InternalError):
        # we don't care if we can't do this -- if we can't, it's
        # because we don't have access to the registry, or because the
        # tools aren't installed.  In either case, the user will have to
        # find them on their own.
        pass
...

scons-master/Scons/Tool/MSCommon/netframework.py

Notes:

• The current source code is largely unchanged from the source code in scons.2.0.0.final.0.
• It does not appear that the find_framework_root and query_versions functions are called from anywhere else in the source tree(s) since 2.0.0.final.0.

netframework module (lines 33-76):

...
# Original value recorded by dcournapeau
_FRAMEWORKDIR_HKEY_ROOT = r'Software\Microsoft\.NETFramework\InstallRoot'
# On SGK's system
_FRAMEWORKDIR_HKEY_ROOT = r'Software\Microsoft\Microsoft SDKs\.NETFramework\v2.0\InstallationFolder'

def find_framework_root():
    # XXX: find it from environment (FrameworkDir)
    try:
        froot = read_reg(_FRAMEWORKDIR_HKEY_ROOT)
        debug("Found framework install root in registry: %s", froot)
    except OSError:
        debug("Could not read reg key %s", _FRAMEWORKDIR_HKEY_ROOT)
        return None

    if not os.path.exists(froot):
        debug("%s not found on fs", froot)
        return None

    return froot

def query_versions():
    froot = find_framework_root()
    if froot:
        contents = os.listdir(froot)

        l = re.compile('v[0-9]+.*')
        versions = [e for e in contents if l.match(e)]

        def versrt(a,b):
            # since version numbers aren't really floats...
            aa = a[1:]
            bb = b[1:]
            aal = aa.split('.')
            bbl = bb.split('.')
            # sequence comparison in python is lexicographical
            # which is exactly what we want.
            # Note we sort backwards so the highest version is first.
            return (aal > bbl) - (aal < bbl)

        versions.sort(versrt)
    else:
        versions = []

    return versions
...

scons-master/SCons/Tool/MSCommon/sdk.py

...
def get_installed_sdks():
    global InstalledSDKList
    global InstalledSDKMap
    debug('get_installed_sdks()')
    if InstalledSDKList is None:
        InstalledSDKList = []
        InstalledSDKMap = {}
        for sdk in SupportedSDKList:
            debug('trying to find SDK %s', sdk.version)
            if sdk.get_sdk_dir():
                debug('found SDK %s', sdk.version)
                InstalledSDKList.append(sdk)
                InstalledSDKMap[sdk.version] = sdk
    return InstalledSDKList
...
def get_sdk_by_version(mssdk):
    if mssdk not in SupportedSDKMap:
        raise SCons.Errors.UserError("SDK version {} is not supported".format(repr(mssdk)))
    get_installed_sdks()
    return InstalledSDKMap.get(mssdk)
...
def get_default_sdk():
    """Set up the default Platform/Windows SDK."""
    get_installed_sdks()
    if not InstalledSDKList:
        return None
    return InstalledSDKList[0]
...

[mwichmann]

One possibility is to deprecate this variable with an eye to removing. It seems pretty clear the intent has been to migrate away from looking at things in terms of Visual Studio but in terms of Visual C++, which more accurately describes what we're calling to build C/C++ code. And there's no particular support for C# and other languages whose projects VS knows how to manage, and msbuild knows how to build. My understanding is that the only direct VS/msbuild knowledge we have is in generating project/solution files.

Another possibility is to put back a few of these as makes sense, in case there's any value to being able to query them, or, put them back in a new information structure, more aligned to how we do things now. The docs are updated for 4.5 to say there's not much available. The added bit looks like this, hopefully that's an adequate description:

Visual Studio 2017 and later do not use the registry for primary storage of this information, so typically for these versions only PROJECTSUFFIX and SOLUTIONSUFFIX will be set.