Add an integration test suite

[shalithasuranga]

Run commands and check created/updated files

[pathange-s]

@shalithasuranga Interested to work on this, if a little more clarification is given on what has to be done. I have written Unit tests for Quizio, a quizzing platform which is built using Node.js for backend.

[shalithasuranga]

Hey.. @pathange-s Thanks for your interest. Please note that this is an integration test suite, so we are not going to refactor and mock existing functions and implement unit tests. Instead, we are going to implement a simple integration test as follows:

When we run npm test create the following operations will happen in a sample test case:

• Calls neu create sampleapp with the Node.js child process API (Global neu installation needed before triggering this).
• Check the command exit code and output. Also, check whether the project has been created.
• If all is OK, mark the test case as passed and continue to the next test case.

Also, we can run all test cases sequentially with the npm test. The goal is to trigger this via the existing Github Actions workflow.

The implementation is very similar to the framework's integration test suite which you can check here. It's great if we can use the same tools (Mocha) and the same project/code structure as the framework spec I think.

Please let me know if you need more details :)

[shalithasuranga]

Btw. @pathange-s seems like the complete implementation of this can be a quite lot of work. So I am not sure whether we can add this too to https://github.com/neutralinojs/gsoc2022 Perhaps, you can let me know your suggestions. We need to test create, run, build, plugins, hot-reload etc too seems

[pathange-s]

Thanks for the details provided. I will look into it. Also, will research on how tough this can prove to be and put my views on adding it to the ideas list. Will revert back in a day.

[pathange-s]

@shalithasuranga back with some data that can help us move towards concluding this. I have written around 70 unit tests using jest for a Node.js backend (Quizio, a quizzing platform where testing is very crucial. I can pull out a gist and send it to you since the repo is yet to be released publicly). Also, I have smoke tested the application using postman. So let me put my views.

• In most cases, we go for integration testing once we are done with unit testing and functional testing on each module of the application. Let me know your views with respect to this application.

• Also, any application in this world will do functional, unit testing compulsory, whereas integration testing will be done only if the modules are dependent on each other.

• Many developers, including me, (what about you 😛 ?) don't write tests, until we have a working feature, or maybe, we don't write any test at all (P.S. In test driven development, we gotta write tests before implementing the feature)

• At the lowest level, we have unit testing (to validate behaviour of individual functions, methods etc). Next level up is integration testing. Also, I suggest that we should done and dusted with unit testing (if necessary for neutralinojs-cli) before writing integration tests. The reason being that integration testing takes input modules that have been unit tested. In integration tests, we test different module's integrability. So, If an integration test fails, it might be a case that the module in itself failed (i.e, unit tests on the module failed). So, if we have no unit tests, we might be struck with checking the module-module dependency but the problem is within the module. Refer the testing pyramid :

This was my understanding. Let me know your views.

So, this seems to be quite a lot of work as you said, I agree on this. How many project ideas and slots can the org have ? Depending on that, we can discuss further on adding it to gsoc2022 repo.

[shalithasuranga]

Hello, @pathange-s Thanks for the detailed explanation about testing. Yeah, traditionally we need unit tests before integration tests. But, we would like to test the CLI using the above-mentioned approach mostly due to the following reasons/requirements:

• We just need to identify feature breakdowns in commits/pull requests -- no need for function level testing/mocking at this point.
• I think implementing both unit and integration makes this minimal CLI project complex and makes the codebase a bit over-engineered at this point.
• Since this is a simple CLI project, we need to focus on each action (command) rather than each function (in code) during GitHub Actions tests

Perhaps, it's neither unit testing nor integration testing. We are going to write a kind of API/specification testing for a CLI. Hope you got the idea ;) And feel free to discuss more details. I've added this idea to the list already.

The idea I am explaining is somewhat similar to these StackOverflow answers:

Regarding project count, I will check further and let you all contributors know :)

[pathange-s]

Cool ! Quite satisfying. Cheers ;) Hope to see this feature implemented soon. 🥂

[pathange-s]

@shalithasuranga, Found an article explaining something similar, give it a look : Integration tests on Node.js CLI

[nimit2801]

Can someone guide me through this issue I've written test cases before in jest? For what exactly we've to write an integration test for?

[shalithasuranga]

Hey.. closing this since we have a test suite now (Implemented via GSoC). Thanks all :tada: