caraya
caraya copied to clipboard
vipm-io
Command Line Interface support for testing folder contents
Caraya cannot be integrated with automation tools for CI CD until it gets command line interface executable for testing all VIs in a folder and returning jUnit xml into a file specified.
It could be integrated. One could create VI which calls all the tests in the folder, and call that VI using CLI - either built in (as in LV 2019), or using G-CLI. I haven't tried CLI in LV 2019, but G-CLI allows to pass parameters into vi, thus report path could be configurable.
This feature is enabled by development for #55. A CLI helper VI will be added to the build.
Next beta build will have this feature enabled.
https://www.screencast.com/t/iV1lHHygrx
The demo shows LabVIEW 2013 calling the Command line interface and instructing LabVIEW 2014 to discover tests in a source folder, run the tests in batch and return a J-Unit compatible XML file as output.
@Primary-Key , feature is available as a pre-release package with version 1.0.0 build 62.
https://github.com/JKISoftware/Caraya/releases/tag/1.0.0.62
Documentation can be found here: https://github.com/JKISoftware/Caraya/wiki/Command-Line-Interface
Is it correct, that this CLI does not return standard output and exit codes? If so, then usage with CI servers like Jenkins will be limited - it will generate test report which could be processed, but in case when something will go wrong it will not return exit code. Have you considered G-CLI tool? It is compatible for LabVIEW >= 2011. Or, you don't want to add such external dependencies to this project?
That is the current implementation. Adding G-CLI could be an option for an add-on package that extends Caraya but I think adding the dependency would make the package much less lightweight. I could be wrong.
I see two options: 1- extend Caraya with an add-on for G-CLI which is independently installable. 2- use a temporary file to let the CI server know that the task is complete. Basically, you can set up the CI server to look for the XML file and fail on timeout if it is not produced.
I would favor option 1. Basically all the hooks are there to do it.
@kosist I chose not to include G-CLI as a dependency in this project, However, I've created an extension package to allow the use of G-CLI. This package depends on G-CLI >=2.1 and Caraya >=1.0.0.67.
The source code and package can be found here: https://github.com/LabVIEW-Open-Source/Caraya-CLI-extension/releases/tag/1.0.0.2
You can see in this snapshot the standard output for tests that pass or fail with custom error codes:
REQUEST FOR COMMENTS
Do we need to extend the CLI support to also be able to run all the tests that are in a project? The workflow would be the same as for folder, except that only the tests defined in the project's lvproj file would be used as a source of tests to execute. This proposed workflow would basically allow to run the exact same tests as those performed by a Project Provider integration or a QD plugin (as suggested in #64 ).
@tmaila made the remark to me yesterday that it would be important to consider the use case of having the automated tests (from CLI) be exactly the same as the ones performed by the developer before one makes a commit. I agree with that sentiment. The current workflow is very obvious from a CI/CD perspective (aka. find all tests under a folder and execute them), but does not operate on the same premise as a developer in the IDE who would like to run all tests without having to first figure out where all the tests are located on disk. It should be as straightforward to run automated tests in the IDE as it is for the CI server.
Adding a way for the CLI to operate on a folder OR a project file would reconcile this gap in the automated testing workflow. CI server could be setup for both workflows. The developer would need to run the batch test utility from the Tools menu or use a QD plugin / projProvider Integration to mimic the same test coverage as the CI process will do.
This is available with pre-release build 72 https://github.com/JKISoftware/Caraya/releases/tag/1.0.0.72.
Still missing a tutorial. Keep issue opened until it does.
As the example CLI vi calls labview.exe with arguments, if I have labview open it does not pass the arguments (as expected as labview is already running). When Allowmultipleinstances = TRUE is added to the labview.ini it solves the problem. Am I missing something? On the video labview was already open the time you ran the vi.
@ImreSzebelledi , you are not mistaken. I think I had LabVIEW 2013 opened and ran the CLI test in LabVIEW 2019...