squad
squad copied to clipboard
Linaro
Boot failure reporting
In LKFT, boot failures are basically not reported for a couple reasons. First, they come in as unique test cases, so they never show up as a regression. Second, it is insufficient to consider a single LAVA job's failure a boot failure.
We would like to report a boot failure when ALL boots fail on a given environment. How could that be possible in SQUAD?
IMHO the change has to start from LAVA. Right now there is no clean way to report a boot success/failure. One way is to report all lava internal test cases in SQUAD (ignored by default). Than 'auto-login-action' would become your boot success indicator. The name would be the same always but it would show up in all test jobs which is not ideal. This was already the case with artificial 'boot' SQUAD creates (based on auto-login-action).
On the other hand I wouldn't want to treat boot as a special case. If there is any other idea it would be nice to hear. IIRC kernelci uses the whole test job to determine whether the boot is successful. So any error in the job excluding infrastructure errors results in reported boot failure.
I think this was addressed in https://github.com/Linaro/squad/pull/937
cc: @stevanradakovic
