Skip tests after clicking “Analyze Code Coverage”.

[ttSpace]

Describe the bug

Python install from Microsoft Store

Steps to Reproduce

1. From Test Explorer, select Analyzer Code Coverage for Selected Tests

   

2. Check Output Window / Tests pane for a warning message indicating that 'coverage' package must be installed.

Expected behavior

Output Window / Tests pane for a warning message indicating that 'coverage' package must be installed. Then it runs successfully.

Additional context and screenshots

Output Info

Building Test Projects
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication8\PythonApplication8\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication8\PythonApplication8;C:\Users\vting\source\repos\PythonApplication8\PythonApplication8\
"C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3056.0_x64__qbz5n2kfra8p0\python3.9.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 51370
Failed to execute tests because the process has already exited.
Connected to socket
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 1 Skipped) run in 3.2 sec ==========

[ttSpace]

In today's build 17.2.0 Preview 3.0 [32323.447.main] , Python Package: 17.0.22080.2 . Install Python from VS. Click "Analyze Code Coverage", as shown below. （Note: Coverage is not installed. ）

Logs

Output for Tests in the Output panel

Building Test Projects
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication13\PythonApplication13\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication13\PythonApplication13;C:\Users\vting\source\repos\PythonApplication13\PythonApplication13\
"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\python.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 56529
Connected to socket
Connection event fired
========== Test run finished: 1 Tests (0 Passed, 1 Failed, 0 Skipped) run in 7.8 sec ==========

[StellaHuang95]

@ttSpace I can't repro this one with Python installed from Microsoft store, could you try this again on the latest version? Thanks.

[ttSpace]

In today's build 17.3.0 Preview 2.0 [32512.412.main], Python Package 17.0.22131.1 . I can repro this with Python installed from Microsoft store .（Note: Coverage is not installed. ）

Logs

Output for Tests in the Output panel

Building Test Projects
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication6\PythonApplication6\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication6\PythonApplication6;C:\Users\vting\source\repos\PythonApplication6\PythonApplication6\
"C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\python3.9.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 57655
Failed to execute tests because the process has already exited.
Connected to socket
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 1 Skipped) run in 3.2 sec ==========

[linette-zyy]

On today's build After clicking "Analyze Code Coverage", The Output Window without a warning message indicating that 'coverage' package must be installed. And no response when click "Show Code Coverage Coloring" from toolbar

[int19h]

When python.exe from the Store is spawned to run the tests, it exits with STATUS_DLL_NOT_FOUND (0xC0000135). Process Monitor records the following failures:

The relevant ones are the last three, which is probing for the same binary, covrun64.dll.

It appears to be a component of the core VS Code Coverage support (i.e. not PTVS). It appears that, when code coverage is enabled, the Python process that we spawn to run the actual tests is hijacked somehow, and code is injected into it that relies on this DLL to function. However, for Store processes, it does not get loaded, probably because they don't use the standard Win32 DLL search path.

The owners of this component might be able to tell more about this, and perhaps provide some workaround or a fix. Until then, I believe the best we can do is disable this feature for Python from the Store.

[AdamYoblick]

@int19h Wow, great digging man :)

When you say "owners of this component", do you mean owners of covrun64.dll on the windows team, or the owners of general code coverage in visual studio?

[int19h]

The owners of covrun64.dll - but that should be owned by whoever owns code coverage in VS. It's not a Windows component - it tries to look it up in System32 only as a last resort, after failing to find it elsewhere.

[jakubch1]

covrun64.dll should be here:

PS C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64> dir


    Directory: C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          6/7/2022   3:35 PM        1084872 CodeCoverage.exe
-a----          6/7/2022   3:35 PM        7167376 covinstr64.dll
-a----          6/7/2022   3:35 PM         401384 covinstrc64.exe
-a----          6/7/2022   3:35 PM        1506192 covrun64.dll
-a----          6/7/2022   3:35 PM        1880488 msdia140.dll
-a----          6/7/2022   3:35 PM         488848 static_covrun64.dll
-a----          6/7/2022   3:35 PM            431 VanguardInstrumentationProfiler_x64.config

Could you please check if you have those files?

[AdamYoblick]

@jakubch1 Yes, I see the files :)

e:\repos\github\PTVS>dir "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64"
 Volume in drive C is Windows
 Volume Serial Number is 621C-D001

 Directory of C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64

05/17/2022  12:36 PM    <DIR>          .
05/17/2022  12:36 PM    <DIR>          ..
05/17/2022  12:36 PM         1,084,384 CodeCoverage.exe
05/17/2022  12:36 PM         7,165,344 covinstr64.dll
05/17/2022  12:36 PM           401,408 covinstrc64.exe
05/17/2022  12:36 PM         1,502,128 covrun64.dll
05/17/2022  12:36 PM         1,880,488 msdia140.dll
05/17/2022  12:36 PM           488,864 static_covrun64.dll
05/17/2022  12:36 PM               431 VanguardInstrumentationProfiler_x64.config
               7 File(s)     12,523,047 bytes
               2 Dir(s)  321,856,397,312 bytes free

The issue isn't that the files are missing, I believe the issue is with the injection of covrun64.dll into the Microsoft store python. Maybe the store python doesn't know where to find the dlls, I'm not sure.

[MarcoRossignoli]

AppStore app dlls resolution is different for security reason and cannot go out “system” folders. This is the reason why covrun is not found using PATH and the search stops inside system32 https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order#standard-search-order-for-uwp-apps

You can disable covrun coverage passing this runsettings inside VS:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
            <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

For autodetect the runsetting the file should be with .runsettings extension and you can instruct VS to find it in this two way https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2022#autodetect-the-run-settings-file In the same page you can find all other way, msbuild prop and manual selection.

[jakubch1]

How to create solution to repro issue? Could you please create one and attach here? I would like to investigate if we can detect that solution doesn't contain any C++ project and disable fully native code coverage.

[AdamYoblick]

@jakubch1 The problem is 100% repro, but it only repros in a specific case. Please see the following steps:

1. Make sure you uninstall all python interpreters on your machine.
2. Install Python 3.9 (or any Python, really) from the Microsoft Store.
3. Install Visual Studio 2022, make sure to include the Python workload.
4. Download the files at https://microsoft-my.sharepoint.com/:f:/p/advolker/EmuKmzWSGEhAiiZ1kIaHnIQBEy3nAJMxiPQmkSJVyueRTA?e=jLZBYV. Open the solution.
5. Click on Test -> Test Explorer
6. Right click on the top node and click on Analyze Code Coverage

Please let me know if you need any more information.

[jakubch1]

Thanks @AdamYoblick for data. I was able to add logic into VS to detect that there are no C++ projects and disable C++ code coverage. This fixed the issue on solution you provided. The change should be available in 17.4 Preview 1.

[ttSpace]

In today's build 17.4.0 Preview 2.0 [32803.438.main] , Python Package 17.0.22215.2 . Python installed from Microsoft store .（Note: Coverage is not installed. ） The output window/test pane without a warning message indicating that the "coverage" package must be installed.

Logs

Output for Tests in the Output panel

Building Test Projects
No C++ projects found in the solution. Dynamic C++ code coverage will be disabled. Static C++ code coverage will be enabled. If report is missing some C++ projects please enable /PROFILE linker option. More information: https://docs.microsoft.com/visualstudio/test/customizing-code-coverage-analysis#static-and-dynamic-native-instrumentation
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication8\PythonApplication8\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication8\PythonApplication8;C:\Users\vting\source\repos\PythonApplication8\PythonApplication8\
"C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\python3.9.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 49227
Connected to socket
Connection event fired
========== Test run finished: 1 Tests (0 Passed, 1 Failed, 0 Skipped) run in 324 ms ==========

[StellaHuang95]

@ttSpace Does this still repro? Seems like it should be fixed by now based on the comment above.

[ttSpace]

@StellaHuang95 In today's build 17.7.0 Preview 2.0 [33705.351.main], Python Package 17.0.23115.2. Python installed from Microsoft store .（Note: Coverage is not installed. ） The output window/test pane without a warning message indicating that the "coverage" package must be installed.

And no response when click "Show Code Coverage Coloring" from the Code Coverage Results panel.

Logs

Output for Tests in the Output panel

Building Test Projects
No C++ projects found in the solution. Dynamic C++ code coverage will be disabled. Static C++ code coverage will be enabled. If report is missing some C++ projects please enable /PROFILE linker option. More information: https://docs.microsoft.com/visualstudio/test/customizing-code-coverage-analysis#static-and-dynamic-native-instrumentation
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication3\PythonApplication3\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication3\PythonApplication3;C:\Users\vting\source\repos\PythonApplication3\PythonApplication3\
"C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\python3.9.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 56287
Connected to socket
Connection event fired
========== Test run finished: 1 Tests (0 Passed, 1 Failed, 0 Skipped) run in 413 ms ==========

[jakubch1]

Message:

No C++ projects found in the solution. Dynamic C++ code coverage will be disabled. Static C++ code coverage will be enabled. If report is missing some C++ projects please enable /PROFILE linker option. More information:

is correct for python solution. What do you expect to see in this scenario? Can you provide simple solution to repro issue?

[ttSpace]

@jakubch1 Please see the following steps:

1. Create a Python Application project.
2. Right-click project, select Add > New Item …, then select "Python Unit Test" followed by "Add".
3. Add test_1.py file with code

import unittest

class Test_test_1(unittest.TestCase):
    def test_A(self):
        self.fail("Not implemented")

if __name__ == '__main__':
    unittest.main()

4. Set Test framework as "unittest" in python project properties page.
5. Open TE with the Test > Windows > Test Explorer menu command
6. Select Analyzer Code Coverage for Selected Tests

Expected behavior

Output Window / Tests pane for a warning message indicating that 'coverage' package must be installed.

After installing the package, click on Analyzer Code Coverage, and click "Show Code Coverage Coloring" from the Code Coverage Results panel. It should look like this:

[jakubch1]

@rchiodo @bschnurr I found string Code coverage file not produced. Please install 'coverage' in PTVS_MSRC repo. What are conditions to show it?

[rchiodo]

Looking in the source, that string is referenced here: https://github.com/microsoft/PTVS/blob/main/Python/Product/TestAdapter.Executor/Common/Services/CodeCoverage.cs#L41

It looks to me like that code is expecting something else to generate an XML file with the coverage results.

The resource string that you referenced is actually referencing this package: https://pypi.org/project/coverage/

So my guess is that you need to install that package into the environment.

[jakubch1]

@rchiodo people are asking why this message is not showing up. It looks like this is not related to projects my team owns.

[rchiodo]

If the message is not appearing, that means it did find the XML file ( I've never worked on this code that I can recall, just looking at the source here). But since there's still an error it probably means the XML file is empty?

[rchiodo]

Or is that the intent? I'm not sure what the change it sounds like you made does to code coverage. Does it still generate an XML file even if there's no native code in the project?

[ttSpace]

In today's build 17.10.0 Preview 1.0 [34521.14.main] , Python Package 17.0.24002.3. Python installed from Microsoft store .（Note: Coverage is not installed. ） The output window/test pane without a warning message indicating that the "coverage" package must be installed. And doesn't show the Code Coverage Results window.

Logs

Output for Tests in the Output panel

Building Test Projects
No C++ projects found in the solution. Dynamic C++ code coverage will be disabled. Static C++ code coverage will be enabled. If report is missing some C++ projects please enable /PROFILE linker option. More information: https://docs.microsoft.com/visualstudio/test/customizing-code-coverage-analysis#static-and-dynamic-native-instrumentation
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication10\PythonApplication10\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication10\PythonApplication10;C:\Users\vting\source\repos\PythonApplication10\PythonApplication10\
"C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\python3.9.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 55246
Connected to socket
Connection event fired
========== Test run finished: 1 Tests (0 Passed, 1 Failed, 0 Skipped) run in 384 ms ==========

[ttSpace]

In today's build 17.11.0 Preview 1.0 [34821.15.main], Python Package 17.0.24110.3 . Python installed from VS.（Note: Coverage is not installed. ） The output window/test pane without a warning message indicating that the "coverage" package must be installed.

Logs

Output for Tests in the Output panel

Building Test Projects
No C++ projects found in the solution. Dynamic C++ code coverage will be disabled. Static C++ code coverage will be enabled. If report is missing some C++ projects please enable /PROFILE linker option. More information: https://docs.microsoft.com/visualstudio/test/customizing-code-coverage-analysis#static-and-dynamic-native-instrumentation
========== Starting test run ==========
cd C:\Users\vting\source\repos\PythonApplication3\PythonApplication3\
set PYTHONPATH=C:\Users\vting\source\repos\PythonApplication3\PythonApplication3;C:\Users\vting\source\repos\PythonApplication3\PythonApplication3\
"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\python.exe" "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON\CORE\visualstudio_py_testlauncher.py" -t test_1.Test_test_1.test_A -r 63323
Connected to socket
Connection event fired
========== Test run finished: 1 Tests (0 Passed, 1 Failed, 0 Skipped) run in 218 ms ==========