galaxy icon indicating copy to clipboard operation
galaxy copied to clipboard

Tool testing: Allow to set test timeout in tools and to expect timeout

Open bernt-matthias opened this issue 1 year ago • 1 comments

Often tool tests need a long time time. In planemo we can now set a general timeout which is is employed by IUC style CI (using 900s). Problem is that this just makes the test fail. We could do better:

What we know after the tool ran successfully for 900s is that the <command> section created a CLI that produced no error.

So my suggestion is that we introduce a boolean attribute expect_timeout and an integer timeout attribute.

  • Assuming that the 900s are just a backup value that allow CI not to fail completely it is useful that tool developers can set a shorter and more useful timeout if they know that the tool will hit the timeout anyway (will save resources)
  • With expect_timeout we may have an option to have a useful test even if the test times out.

Some more initial thoughts:

  • Sometimes the tool may have even produced output that could be checked.. but maybe we should not count on it.
  • I guess there will be specific exit code?

bernt-matthias avatar Sep 21 '22 09:09 bernt-matthias

In other words: if the tool can parse the command line and maybe the input data within the timeout then we already know quite a lot about the correctness of the Galaxy tool. The underlying tool may still fail if we would give it more time ... but this might be a problem of the underlying tool in most cases / often.

bernt-matthias avatar Sep 21 '22 09:09 bernt-matthias