galaxy
galaxy copied to clipboard
Tool testing: Allow to set test timeout in tools and to expect timeout
Often tool tests need a long time time. In planemo
we can now set a general timeout which is is employed by IUC style CI (using 900s). Problem is that this just makes the test fail. We could do better:
What we know after the tool ran successfully for 900s is that the <command>
section created a CLI that produced no error.
So my suggestion is that we introduce a boolean attribute expect_timeout
and an integer timeout
attribute.
- Assuming that the 900s are just a backup value that allow CI not to fail completely it is useful that tool developers can set a shorter and more useful timeout if they know that the tool will hit the timeout anyway (will save resources)
- With
expect_timeout
we may have an option to have a useful test even if the test times out.
Some more initial thoughts:
- Sometimes the tool may have even produced output that could be checked.. but maybe we should not count on it.
- I guess there will be specific exit code?
In other words: if the tool can parse the command line and maybe the input data within the timeout then we already know quite a lot about the correctness of the Galaxy tool. The underlying tool may still fail if we would give it more time ... but this might be a problem of the underlying tool in most cases / often.