Even simpler way to define tests (by only calling Assert)

[jimkring]

Use Case

As a developer working on a project, a simple workflow I would like is:

• Pick a folder that contains all of my tests. e.g. >> "source/tests"
• Create a new test VI and save it as "Test Feature X.vi" inside the "source/tests" folder
• Tell Caraya to run all the tests in the "source/tests" folder

Wash, rinse, and repeat. Success!

"Problem" (ok, it's more of an inconvenience)

When creating my tests, it sort of feels like a minor-but-tedious pain to have to drop both the Define Test and the Assert Vis.  I'd like to not have to drop any Define Test VIs.

Ideas

Maybe we could change the behavior of Caraya in some way, such that the presence of an Assert defines the caller as being a test.

Here are some different ideas:

• Change the whole framework to assume that any call to "Assert" also means Define Test.  Let everyone know that Caraya is a test framework and no longer an assertion framework. Boom 🤯
  • Note: Caraya was originally developed to be both an Assertion Framework and a Test Framework, and so the Assert VIs themselves can be called outside of testing.  However, most people do not actually use Caraya for run-time assertions in their applications (does anyone?).
• Add a new Boolean input to Assert called "Define Test": if TRUE then it calls Define Test behind the scenes.
  • Question: would it make sense to have TRUE be the default value? Then what happens for users who like the old Assertion behavior?
• Create a new set of Assert VIs (and a new PolyVI that makes it easy to call them) that also calls Define Test behind the scenes.  Maybe deprecate the old Assert VIs and let people call them if they like.
• Maybe we could add a global setting in Caraya called "Treat Assertions as Tests" that toggles this behavior. We could leave the default setting as FALSE in the next release but make it easy to set to TRUE.
  • In the future, we could change the default setting to TRUE and warn users (who like the Assertion style use-case) that the default mode is changing and they will need to be sure to set "Treat Assertions as Tests = TRUE" in their applications to get the old behavior.
  •  "Run Caraya Tests in Folder" could have an input "Treat Assertions as Tests" that sets this setting before running all the tests.
  • The Caraya GUI could allow users to set/change this setting easily via the GUI when doing a "Run all tests in project"

I'm curious what others think about this...

[jimkring]

[update] I added another bullet above:

• Maybe we could add a global setting in Caraya called "Treat Assertions as Tests" that toggles this behavior. We could leave the default setting as FALSE in the next release but make it easy to set to TRUE.
  • In the future, we could change the default setting to TRUE and warn users (who like the Assertion style use-case) that the default mode is changing and they will need to be sure to set "Treat Assertions as Tests = TRUE" in their applications to get the old behavior.
  • "Run Caraya Tests in Folder" could have an input "Treat Assertions as Tests" that sets this setting before running all the tests.
  • The Caraya GUI could allow users to set/change this setting easily via the GUI when doing a "Run all tests in project"

[francois-normandin]

It could work, but I think it introduces a few quirks that we'd need to change:

• First, you'd lose the ability to organize the test results in categories (Test Label) and would instead be forced to organize by caller hierarchy or by flattening all tests... although you'd possibly encounter name collisions in that latter scheme.
• Second, you'd need a new method for creating reuse components (that is, encapsulating assertions that will be called multiple time by different labels, or called in a loop to assert multiple conditions)
• Third, you couldn't use the "Define Test" alongside conditional disable structures to have assertions in the built application, and tests in the development environment.

I'm sure there are more "effects" that might appear, but perhaps we can have our cake and eat it? Why change the current behavior when we can simply create a new first-class citizen method that would be a "Test Unit" and act in the way you describe?