robotframework
Enhancement request: The ability to set suite timeouts
Background:
The test suites I manage can potentially unpredictably run for extremely long times. This causes issues with our CI/CD system, as it can cause the pipeline to hang and work not to get completed on time.
Solution request:
I would like to add a new entry in the *** settings *** section of a suite called suite timeout. It should behave similarly to the test timeout entry, but the run duration is calculated as an aggregate of all the tests in the suite.
When the run duration is higher than what is set in suite timeout, the current tests + all following tests should fail.
Additionally, the ability to set a default value for suite timeout from the command line would be ideal.
Alternatives considered
Setting timeout values on the CD pipeline:
- Timeout is based per stage, not on a per suite basis, (eg, All tests tagged with "smoke" should run within 10 minutes)
- You lose the logs as the pipeline terminates the robot process early.
Test timeouts
- As the spikes in test duration are unpredictable, you either make the test timeout "too tight" or "too lose". The First will not help with pipeline backups, and the second will lead to false positives, which is a maintenance burden.
Custome visitor API implementation
- Some initial testing indicates that the visitor API does not enable the ability to mutate test cases, which is required to fail them early.
