vscode-powershell
vscode-powershell copied to clipboard
PowerShell
WIP: Setup unit test code coverage using `nyc`
So @rjmholt nyc does not currently work on Windows. There's a myriad of bug reports about it, and they haven't pushed a new release in over a year. It is very sad.
https://github.com/istanbuljs/nyc/issues/1381 repros on my Surface, without any symlinks. Looks like they messed up path.join vs path.resolve and the casing that's returned for root drive letters causes the case sensitive globs to fail.
I'll split out a couple things from this and leave it WIP I guess. Sucks, I can't find another tool for JavaScript code coverage. @TylerLeonhardt do you have any suggestions?
Ok so at this point we are punting again. This branch produces a coverage report on my local macMini:
----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
All files | 45.71 | 24.09 | 34.78 | 49.93 |
src | 56.54 | 32.31 | 54.86 | 61.05 |
debugAdapter.ts | 82.6 | 64.28 | 63.63 | 81.81 | 36,46,54-55,60-61,68,96
languageClientConsumer.ts | 58.33 | 25 | 50 | 71.42 | 19-22
logging.ts | 45.28 | 30.76 | 44.82 | 48 | 49-58,93-133,154-155,157-168
main.ts | 86.2 | 46.15 | 60 | 86.2 | 197-217
platform.ts | 35.15 | 18.37 | 55.81 | 40 | 45,48-49,118-131,151-152,174-179,198-211,221,234,247,263-288,303-464,469
process.ts | 87.93 | 53.57 | 71.42 | 87.27 | 52,131,149,175,204-210
session.ts | 53.29 | 39.2 | 45.83 | 53.29 | ...8,196,216-245,265,290-320,337-338,345-365,393-409,420-421,450,485-499,535-539,557-559,590,606-624,682-686,692-700,717-786,794-796
settings.ts | 95.06 | 82.14 | 86.66 | 94.23 | 287,293,301
utils.ts | 61.62 | 25 | 57.69 | 77.35 | 21,33-38,44-49,86,106,113-114
src/controls | 24.52 | 0 | 0 | 24.13 |
checkboxQuickPick.ts | 24.52 | 0 | 0 | 24.13 | 27-96
src/features | 38.26 | 17.59 | 24.53 | 42.4 |
CodeActions.ts | 35.71 | 0 | 16.66 | 35.71 | 14-44
Console.ts | 29.19 | 16.66 | 17.77 | 38.82 | 89-193,207-250,255-264,291-292,299
CustomViews.ts | 31.54 | 27.27 | 15.78 | 45.65 | 18-25,33-64,81-137,169-173,204-233
DebugSession.ts | 32.81 | 13.76 | 22.72 | 32.28 | 61-111,165-168,185-212,251-298,348-372,410-411,425-510,540-541,555-630
Examples.ts | 71.42 | 100 | 33.33 | 71.42 | 15-23
ExpandAlias.ts | 47.05 | 0 | 20 | 47.05 | 19-45
ExtensionCommands.ts | 20.43 | 5.63 | 4.54 | 21.21 | 54-86,191-227,239-625
ExternalApi.ts | 93.1 | 62.5 | 85.71 | 92 | 66,71
FindModule.ts | 28 | 0 | 10 | 28 | 24-93
GenerateBugReport.ts | 42.85 | 0 | 25 | 40.74 | 18,34-118
GetCommands.ts | 65.9 | 77.77 | 57.14 | 65.11 | 35,48-52,64-65,69,77-82,132-139
HelpCompletion.ts | 41.17 | 15.78 | 44.44 | 40 | 45,63-69,95-128,146,154-193
ISECompatibility.ts | 54.83 | 37.5 | 50 | 63.15 | 34-40,51-53,60-61
NewFileOrProject.ts | 33.33 | 4 | 11.76 | 33.33 | 20-52,65-167
OpenInISE.ts | 50 | 0 | 33.33 | 50 | 13-32
PesterTests.ts | 75.86 | 54.54 | 61.53 | 75.86 | 33-44,61-65
RemoteFiles.ts | 68.42 | 40 | 55.55 | 72.22 | 52,65-70
RunCode.ts | 51.21 | 35.29 | 25 | 60 | 23-52,63
ShowHelp.ts | 43.75 | 0 | 33.33 | 43.75 | 18-35
UpdatePowerShell.ts | 34.09 | 8.97 | 27.77 | 42.85 | 33-36,47-48,53,67-71,105-190
----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
nyc
│ 56.98 % Lines
└─┬ /usr/local/bin/node /Users/andschwa/src/vscode-powershell/out/test/runTests.js
│ 56.98 % Lines
└─┬ /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/MacOS/Electron --disable-extensions ./test --no-sandbox --disable-workspace-trust --extensionDevelopmentPath=/Users/andschwa/src/vscode-powershell --extensionTestsPath=/Users/andschwa/src/vscode-powershell/out/test/index --crash-reporter-id 711b0fb3-33c9-4e09-a2f3-289124f3fa1e
│ 56.34 % Lines
└─┬ /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/MacOS/Electron
│ 56.34 % Lines
└── /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Renderer).app/Contents/MacOS/Code - Insiders Helper (Renderer) /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-fork --type=extensionHost --skipWorkspaceStorageLock
56.34 % Lines
And so it should work in CI, but it doesn't. It only runs if we set use-spawn-wrap to true (which is being deprecated anyway) and then it doesn't actually produce any results because it fails to analyze files touched by the subprocesses the Code extension test suite executes.
I am far from the only person struggling with nyc (which is unfortunate given it's the code coverage tool for JavaScript). Here's a prior Code C# / OmniSharp extension maintainer having issues: https://github.com/istanbuljs/nyc/issues/728
And despite eventually getting it working, the implementation in that extension's CI was eventually removed: https://github.com/OmniSharp/omnisharp-vscode/pull/4506#issuecomment-949843807
To quote the maintainer directly:
I remember there being a lot of difficulties getting it to work as expected when moving from the vscode to the vscode-test package. The effort to get it working wasn't justified by the value we got from it.
While an overview of the Code Python extension looks like it uses nyc (and indeed I followed their setup initially) it turns out that they have their own test runner, and thus are able to execute their JavaScript unit tests directly on Ubuntu without instantiating Code. Indeed, they do not even install the @vscode/test-electron package which most extension's use to launch Code (and then it launches the tests within Code, hence the subprocesses). So it's not actually a useful example for us.
Now theoretically, if we want to take one more try at this, we could pre-instrument the codebase dynamically after the test Code instance is launched by having Mocha use Babel to transpile the JavaScript and babel-plugin-istanbul to instrument the code, and then maybe nyc would work. But there's no guarantees and it seems, in a word, difficult. This was suggested here.
Also here two Code extension with coverage examples, but they're out of date as far as I can tell:
- https://github.com/codecov/example-typescript-vscode-extension
- https://github.com/jedwards1211/vscode-extension-skeleton
I don't think I'll get back to using Istanbul, it's sooo out of date. We'll do this another way.