Provide a way to use python environment when using cmake-presets

[mkilivan]

Brief Issue Summary

I would expect vscode-cmake-tools to invoke the cmake binary within the python virtual environment that is the currently chosen interpreter by the python extension when using cmake presets.

Currently, any python dependencies in a cmake workflow must be installed in the global python environment when using cmake-presets.

[bobbrow]

This extension currently has no integration with the python extension so it doesn't know about your python virtual environment. @esweet431 is this a scenario you are familiar with, or that Kitware has mentioned to you?

[bobbrow]

Related discussion: https://github.com/microsoft/vscode-cmake-tools/issues/1652

[esweet431]

This isn't something I'm familiar with. For my own understanding, if you were working from the command line without VS Code, would the workflow be:

1. Set-up python environment
2. Invoke CMake from that environment

...and you want a way for cmake-tools to invoke CMake from the environment set by the python extension?

[mkilivan]

Yes @esweet431. The version of CMake in the system path and in the python environment is different. It's a remote ssh project and currently, the only option is setting up the python environment from VS Code terminal and running CMake from there.

It would be nice to invoke CMake from the selected python environment instead of from the global system path.

The following is the Output window of the CMake/Build. And you can see cmake invoked from the system path.

[proc] Executing command: /usr/local/bin/cmake ...

Would it be possible to use python.pythonPath in settings.json and activate the python environment before invoking CMake?

[esweet431]

Would it be possible to use python.pythonPath in settings.json and activate the python environment before invoking CMake?

This is a feature request for cmake-tools to integrate with the python extension. I believe this is possible but I'll let @bobbrow triage and confirm.

As a workaround, we're tracking #21619, which asks presets to support sourcing an environment script before configure. This is related and would potentially provide a workaround, but it would still require you to duplicate the environment set in the python extension in a script called by the preset. The solution of integrating with the python extension will be specific to cmake-tools.

[craigscott-crascit]

Hi folks, CMake co-maintainer chiming in here. This isn't really a problem that can or should be solved on the CMake presets side. I'll try to illustrate through a real scenario from one of my consulting clients, but it would be a relatively common scenario more broadly.

Let's say a project produces python modules that have compatibility requirements that require a specific python version. This typically needs to match the major.minor version, not just major. It uses pybind11, which ends up calling find_package(Python) or find_package(PythonInterp) to find the python installation to build against. The project may also run the python interpreter to perform various tasks during CMake configuration. This is the motivation for needing a specific python environment to be provided to the CMake configuration step.

Company X wants to make it easy for its developers to get started with the project. They provide a CMakePresets.json file which defines the commonly used configurations, including the ones used in CI jobs. Developers may be getting their python version via different mechanisms, such as a pre-prepared docker image, conda, a system package manager, manually manipulating the PATH, and so on. Each of these have their own unique needs regarding how the environment should be set up. Some should only be done by invoking a particular command to load a shell with the environment prepared, others might not have such a command.

These differences mean this isn't something that a preset can capture without assuming a particular python environment provider. Some presets may be intended to span across platforms where different python virtual environment providers would be expected, so you would have to duplicate presets just to account for that if you went down that path. Even then, having to account for each different way someone may want to provide their environment doesn't scale. Some users would get things as set up by their IT department. Other users may need to experiment with different python versions as they take on new package dependencies, etc. Such variability won't fly with projects that already have many presets (consider the combinatorial explosion of build presets already identified as being problematic, which duplicating configure presets would only exacerbate further).

Pushing the problem to user space and asking users to put this information in their CMakeUserPresets.json file is not really a solution either. It still doesn't properly handle the scenario where a command needs to be run to set up the environment properly. Many users have little to no familiarity or interest in that, their focus may be on developing some mathematical algorithm, not futzing around with software build system configuration. More importantly, VS Code is already aware of the available python environments thanks to the python extension. CMake presets shouldn't need to duplicate that information. VS Code has all the information it needs to be able to execute CMake commands in an environment that can be selected by the user from the available python environments without them really having to do much. This choice is already available to them for starting an arbitrary terminal.

Even if you disagree with the above, consider the case where there are no build or test presets defined, only configure presets. If any custom commands defined by the build need to use python, the IDE still has to prepare the python environment without help from presets. Same situation if tests run python (in my real scenario, some of the tests run pytest under the hood).

It is also worth pointing out that this isn't without precedent in VS Code. It already prepares an environment in which to run CMake and the build tool by interrogating things like the preset's architecture, toolset and cacheVariables fields (the latter for CMAKE_C_COMPILER and CMAKE_CXX_COMPILER). It doesn't seem like a major stretch to apply those to a shell prepared by a python environment instead of some default shell.

I probably haven't expressed the above scenarios all that clearly, but hopefully at least I've sketched out enough of some real problems that it is clear this is better solved on the VS Code side. It would result in a better user experience, in my opinion.

[jk000]

In addition to getting an environment from the Python extension, and as mentioned in #1652, it would be great to have a way to load arbitrary, and even multiple environments.

For example:

$ source /opt/rh/devtoolset-10/enable        # Red Hat Developer Toolset
$ micromamba activate test_env               # Conda environment without a Python installation
$ source ~/.venv/bin/activate                # Python virtual environment (could get from Python extension or manually)

[TBBle]

As a workaround for detecting the wrong PythonInterpreter (system Python install instead of the active virtual env), if you open a terminal in VSCode, the selected Python environment is activated, and you can just copy the CMake command-line run by the CMake attempt and run it there. You'll have to remove any detected Python interpreter from CMakeCache.txt first, but once that's done, rerunning CMake through CMake tools won't redetect Python, as it'll store the venv-based interpreter in CMakeCache.txt and use it in future invocations.

This doesn't solve the original poster's issue that they have cmake (or any other desired scripts) in the Python venv's Scripts directory and expect it to be found by normal PATH lookup though.

Based on the above workaround, if the cmake-tools behaved as if they were run in a normal VSCode terminal, then any other extensions that affect the terminal environment (like the Python extension auto-sourcing your selected venv) would work naturally, without needing specific integration. That would also do the right (expected) thing with PATH lookups, since venv activation (and other PATH-affecting extensions) would have been applied already.

I'm not sure if that's actually feasible though, and also possibly doesn't make sense in remote session; maybe it does make sense, if the Terminal system already does the right thing in the remote session case.

[lpapp]

I am also unable to build my project using Visual Studio with both the cmake tools and python interpreter.

Would be good to get extensions co-operating better as if they were built-in. Otherwise, the idea of extensions can be disadvantageous.

[craigscott-crascit]

For others who are getting stuck behind this issue, a possible workaround in the interim is to write your own wrapper script around the cmake and ctest executables. In those wrapper scripts, you would prepare whatever environment you need to use before forwarding on the command line arguments to the real cmake and ctest binaries. You can override the cmake and ctest used by VS Code in its settings (look for Cmake path and Ctest path), where you would point those at your wrapper scripts instead of the real binaries. I have used this successfully to run both with a specific conda environment, for example.

[lpapp]

A simpler workaround that I found is just to build from the terminal. But it drives me away from using an IDE like VSCode if I cannot use cmake and python out of the box simultaneously which are so common today for C++ projects.

I was hoping to use an IDE to free me up and make me more efficient, and not to force me writing further scripts (especially custom to one particular IDE unlike CMakeUserPresets.json) just to be able to build and run a project in an IDE like VSCode. Sorry if this sounds harsh, do not mean to be.

With the volume of 15-20 work trees (and 3 different OSes), project specific scripts are suboptimal as you would need to maintain them across 15-20 work trees in case you change anything in them. I would not personally go down that route in my case. It does not appeal to me considering that I can just build in the terminal.

I would build in the IDE if I could set the python interpreter for once and all, but looking at this report, this has not gained any interest over the past year to get a fix :(

If the extension architecture is getting in the way, I would suggest to create an overarching C++ extension which includes all the common tools of today's world, like cmake, C++, git, python, etc, to have a choice for those who just want to get things working and not to prematurely micro-optimise with extension cherry-picking.

[elupus]

Just ran into this too. It's easy to handle sourcing a venv from command line. But not being able to use the selected venv from vscode when configuring cmake is really annoying. vscode will not defere configuration, it will try to do it automatically, which will fail if run without proper dependencies installed from for example venv.

[fgrie]

For others who are getting stuck behind this issue, a possible workaround in the interim is to write your own wrapper script around the cmake and ctest executables. In those wrapper scripts, you would prepare whatever environment you need to use before forwarding on the command line arguments to the real cmake and ctest binaries. You can override the cmake and ctest used by VS Code in its settings (look for Cmake path and Ctest path), where you would point those at your wrapper scripts instead of the real binaries.

Tried this in the past - VSCode just complained not to be able to find cmake.exe, unfortunately. How exactly did you set it up to succeed?

Edit: Just tried it, the following settings.json did not work:

{
    "cmake.cmakePath": "${workspaceFolder}/tools/cmake_wrapper.ps1"
}

Error message:

Bad CMake executable: 
c:\Users\Username\Path\tools\cmake_wrapper.ps1". Check to make sure it is installed or the value of the "cmake.cmakePath" setting contains the correct path

Win10, VSCode 1.78.2

[benmcmorran]

Thanks for all the feedback folks. Adding this support in the CMake Tools extension is currently blocked on better support for extensions setting environment variables in VS Code, which is tracked by https://github.com/microsoft/vscode/issues/152806.

[craigscott-crascit]

@fgrie Try a batch file rather than powershell.

[dmitry-kabanov]

For others who are getting stuck behind this issue, a possible workaround in the interim is to write your own wrapper script around the cmake and ctest executables. In those wrapper scripts, you would prepare whatever environment you need to use before forwarding on the command line arguments to the real cmake and ctest binaries. You can override the cmake and ctest used by VS Code in its settings (look for Cmake path and Ctest path), where you would point those at your wrapper scripts instead of the real binaries. I have used this successfully to run both with a specific conda environment, for example.

I walked this path proposed by @craigscott-crascit and it worked for me. I'd like to share the details in case you work in Linux (pretty sure it will work in macOS as well):

1. Create cmake_wrapper.sh in the project root directory (workspaceFolder in vscode terminology)
2. This is what the script should look like:

#!/usr/bin/env zsh

HERE YOU NEED TO COPY ALL THE INITIALIZATION CODE FOR CONDA
FROM YOU ~/.bashrc or ./zshrc
OTHERWISE IT WILL NOT WORK

CUR_DIR="$(realpath $(dirname $0))"
source "${CUR_DIR}"/autoenv-enter.zsh

cmake $@

Note that the actual environment definition for me is in the file autoenv-enter.zsh (it is executed automatically in a terminal), so to avoid duplication, I just source this file. 3. Make the script executable: chmod u+x cmake_wrapper.sh 4. In the workspace settings window in vscode, find cmake path field and fill it in: path-to-the-workspace/cmake_wrapper.sh. Note that if you use a worksapce variable (like this: ${workspaceFolder}/cmake_wrapper.sh, then CMAke tools strangely do not evaluate the variable initially, but evaluate it after failed attempt, so I just specify the actual path by hand. 5. Restart vscode and make sure that CMake runs correctly with the provided environment.