[QA] Allow early fail in drone CI

[saw-jan]

Pre-submission Checks

• [X] I checked for similar issues, but could not find any. I also checked the closed issues. I could not contribute additional information to any existing issue.
• [X] I will take the time to fill in all the required fields. I know that the bug report may be dismissed otherwise due to lack of information.

Describe the QA issue

During the drone CI build, ctests pipeline or Squish tests pipelines might fail. If one pipeline fails then we may want to stop the build like we do in other oC repos (web, ocis). Also, we may want to stop the Squish test run and fail the test pipeline if one of the Squish tests fails. With this we can get early notification/feedback about the failure and don't have to wait till the build is finished (or don't have to manually stop the CI build)

• early notification/feedback on failure
• prevent unnecessary resource usage
• release Squish licenses early

In order to disable the above-mentioned behavior we can implement the use of full-ci concept like in other repos. With full-ci, the pipelines will not fail early even there are pipeline failures and all Squish tests will run regardless of some test failure.

If we want to implement this then the following will be the tasks to do:

• [ ] Early fail the build if one of the pipelines fails
• [ ] Allow abort on failure for Squish tests (using --abortOnFail arg)
• [ ] With full-ci in the PR title and nightly builds, the above two behavior must be disabled.

CC @individual-it @TheOneRing @fm

[TheOneRing]

I agree that we should abort if ctest failed. Sadly the squish tests are to unreliable at the moments to abort if a single one failed.

[saw-jan]

It would really be very nice to early fail the CI build because currently, we run two Squish test pipelines: for oC10 and oCIS which directly affect the Squish license consumption. This will cause other CI builds to wait for long. But if we really want to run CI build without causing early fail then there will be an option to do so. For example, in other repos, we use full-ci in the PR title and run CI build completely regardless of the failure.

[TheOneRing]

Hmm thinking about that special case it doesn't sound too bad.

[saw-jan]

@TheOneRing worth implementing it?

[saw-jan]

The --abortOnFail option is available only when using --testsuite option but does not work with --testcase.

--abortOnFail If the --abortOnFail option is specified, squishrunner will terminate the suite execution as soon as a failed test case is detected.

This means that if we have test cases 1, 2, 3 and 4, and test case no.2 failed then test cases 3 and 4 will not run. Also if the fail was in the first scenario of test case no.2, other scenarios of test case no.2 still continue to run.

[saw-jan]

• [ ] Early fail the build if one of the pipelines fails

this is not possible with the current drone setup