Test runners in vscode are much slower than pycharm

[dickreuter]

Running tests in pycharm are much faster than in vscode. This poses a significant disadvantage for vscode for any serious Python project that has a large test suite.

[eleanorjboyd]

Hello! Would you be able to describe your use case further so we can look into the issue? If you could include an outline of your project structure (or example project which you see this behavior) and why it feels slow we can go from there. Thanks!

[dickreuter]

The best way to see how slow the native test explorer is, just run all pytests tests of any project. Then get the Python test explorer for Visual Studio code extension, and run all the tests with that extension. You will see that there's a speed difference of more than 2x. The native test explorer is much slower.

For example try this project and set up pytest: https://github.com/dickreuter/Poker

Any other project will experience the same.

[eleanorjboyd]

By native you mean without the python extension? @connor4312, what are you thoughts here?

[dickreuter]

By native I mean without the Test Explorer UI, which is abandoned and recommends switching to vs code native testing. https://marketplace.visualstudio.com/items?itemName=hbenl.vscode-test-explorer

[eleanorjboyd]

understood- we have a few thoughts on improving performance I will investigate this further. Thanks

[m-ad]

I can confirm the issue: I have a Python project where running the pytest suite takes 0.2s - 0.3s in Pycharm, whereas VSCode's native Test Explorer takes between 5 and 7 seconds, that's a slowdown of more than a factor of 25. Interestingly, running the tests in a Windows terminal (not in VSCode), the tests take around 1.0s to 1.8s. That's slower than in Pycharm, but still siginificantly faster than VSCode. Don't know what kinds of optimizations Pycharm does here, but it would be a start if VSCode wouldn't be slower than the terminal.

[MRiabov]

Folks, oftentimes vscode's pytests are not working at all. When it is stuck like so it won't even turn itself off.

And the tests consume much more memory. For example. I have to run tests in batches of 1-3 because running 5 clogs the memory so much I have to either wsl --shutdown or wait for 30 minutes while the caching to disk processes itself.

Whereas calling pytest -k ... from command makes all tests run in under 20 seconds. What are you doing under the hood that creates so much trouble?

[theypsilon]

I experienced the slowness too, in my case is 600-800ms in pycharm vs 2.7-5secs in vs code. I run over 500 unit tests of this OSS project: https://github.com/MiSTer-devel/Downloader_MiSTer

I run all the tests under src/test/unit and src/test/integration that start with test_* of the wip branch (around 10 of them are supposed to fail).

Here is the VS Code settings in case it helps:

{
  "python.testing.unittestEnabled": true,
  "python.testing.pytestEnabled": false,
  "python.testing.nosetestsEnabled": false,
  "python.testing.env": {
    "PYTHONPATH": "${workspaceFolder}/src"
  },
  "python.testing.cwd": "${workspaceFolder}/src",
  "python.testing.unittestArgs": [],

  "python.analysis.extraPaths": ["./src"],
}

[crushndent]

I have a much larger test base. I'm seeing 4 minutes in vscode vs 1.5 minutes in pycharm. 500+ test cases. This is on native linux.

[dickreuter]

Test explorer in vscode is still highly inferior to pycharm, especially also in the discovery. One wrong import and the whole test explorer breaks down. This is much better handled in pycharm where tests that are not affected can still be discovered.