pnp
💡 [Feature]: for the love of god let's start writing tests
🎯 Aim of the feature
It's kinda embarrassing but yes we do not have unit tests as of now. What initially started as a small VS Code extension oriented around ACEs and Viva Connections turned to this massive extension that extensively enhances VS Code with lot of SPFx development features, now even extends GitHub Copilot. As we grow and grow we need to make sure we increase the quality and maintainability of our work. As first step we need to add unit tests. The aim of this issue is to prototype this and kick of with a small portion of unit tests for a single provider or service.
After further researching and prototyping this functionality I decided the best approach would be to go with kind of integration tests that will allow us to test the extension behavior and functionality which is more broad. I checked few approaches and investigated the one recommended by the VS Code team
Together with Extension Test Runner extension allows to support debugging this in real life which is cool. We may automate this using vscode-test CLI .
Short plan:
- [x] Create prototype with a single tests to check if extension is installed
- [x] Prototype a solution to run tests in GitHub Workflow
- [x] Create sub or follow up issues with suites (groups) for integration tests that we may add to the extension
- [x] Lets wait for the first two sub issues to be done and see if we learn something more and then add more issues for other part of the extension
Hi @Adam-it. I did some research on this and see that it is possible to do some unit tests with Mocha or with a third party vscode-extension-tester that appears to be more suited to test aspects of an extension. What are your thoughts on approach? I would like to work on this implementation starting with prototype.
Hi @Adam-it. I did some research on this and see that it is possible to do some unit tests with Mocha or with a third party vscode-extension-tester that appears to be more suited to test aspects of an extension. What are your thoughts on approach? I would like to work on this implementation starting with prototype.
Thanks @ervingayle for your initiative 👏👍. Unfortunately this issue is still not ready to be taken (it is not labeled as 'help wanted') and I am still thinking and prototyping on the best approach to go about this task. TBH for the last couple of months I haven't found time fot this yet 🫤, there is always so much to do 😉. That being said I am 100% open for feedback and suggestions. Till now I haven't picked an approach yet so if you had anything specific in mind or would like to point me to an example or article or tech you would use I would really appreciate it. Till now I considered just writing standard unit tests using Mocha as I have some experience in it and this would work perfectly for services like CliActions. On the other hand many parts of this extension is not ready for testing, like those that extend VS Code UI with web views, and here I wanted to do something like integration testing that would run the extension based on a scenario but top priority is to make it work in GH Actions.
@ervingayle do you want to share your research?
Hi @Adam-it. I understand. I see that the Microsoft guidance is scarce and does not go into much detail about what can be tested even though it says that the components they offer are based on Mocha. The samples that are offered does not show the full extent of the unit tests. This is what my comments are based on thus far: Microsoft Docs. However, I saw a reference to this: vscode-extension-tester and I did some research on this and looked through the issues in to repo as I know the vscode-viva extension has webviews and this test suite seems to support testing this similar to how Jest has support for screen and expect xyz to be in the document/on screen. Here is the approach for WebviewView's. What do you think?
I did some research around that and it seems using @vscode/test-cli @vscode/test-electron
and Extension Test Runner extension to support debugging this in real life.
We may automate this using vscode-test.
This installs local version of VS Code and run all tests specificed in a separate instance of VS Code allowing us to check the extension functionality. This approach will allow us to use vscode API instead of mocking it and will allow us to create integration tests which is more flexible and what we aim
Now that both issues are done we should revise what we learned and based on that create more issues with creating tests for our extension. It would be best if our tests would check behavior not functionality but I am opened to any discussion. Also I found this kind of package that could help us with testing behaviors https://github.com/redhat-developer/vscode-extension-tester
@Saurabh7019 since you did majority of work on those issues I was wondering what is your take on that? What kind of issues would you add and what you would like to test?
Oh I had overlooked this.
If I remember it correctly, I was getting ESM related errors as soon as CommandPanel was imported in test suites. Some package was using require('..') inside ESM.
It is still hard for me to write issues with exact details since some research is needed, but I think we can start with the following issues and continue the research as part of them
Add integration tests for:
- Account Tree view - verify account tree structure, including sign-in/sign-out options, account label, SharePoint URLs, Entra app registration
- Action Tree view - verify Action tree view links
- Sample Gallery - verify loading from sample JSON, filters, etc.
- Environment Tree view - verify both tenant and site app catalog section and options
what do you think?
@Saurabh7019 ok clear. Indeed adding tests for VS code extensions is not that easy. I say lets start with a single thing from the points you suggested and focus on that asspect and investigation. We have already a bunch of higher priority thing we want to add to this tool but lets try not to forget about those tests. I suggest we continue with creating and issue for adding tests for the sample gallery functionality. This might be quite interesting since this is based on webview feature so we will need to figure out how works testing this functionality. Do you want to add an issue for that one?