gramineproject
LTP parser doesn't consider standard error output to determine test pass/fail
Description of the problem
Current LTP parser doesn't consider output from standard error and hence any error messages over there are completely ignored.
https://github.com/gramineproject/gramine/blob/master/LibOS/shim/test/ltp/test_ltp.py#L117
For ex: erorr message in standard error for waitpid03 was not considered and the test was reported as a PASS. https://localhost:8080/job/graphene-20.04/2260/testReport/LibOS.shim.test.ltp/test_ltp/test_direct___test_ltp_waitpid03_/
<testcase classname="LibOS.shim.test.ltp.test_ltp" file="LibOS/shim/test/ltp/test_ltp.py" line="185" name="test_ltp[waitpid03]" time="0.129">
<system-out>
waitpid03 1 TPASS : Got correct child PID
waitpid03 2 TPASS : Condition 2 test passed
</system-out>
<system-err>[P23:T23:waitpid03] error: Sending IPC process-exit notification failed: -13
</system-err>
</testcase>
Steps to reproduce
Expected results
Actual results
Gramine commit hash
What's the issue here? Why would writing anything to stderr be considered an error? Debug logging is often done this way.
Also this seems like https://github.com/gramineproject/gramine/issues/21 and in this case it's benign issue.
For ex: erorr message in standard error for waitpid03 was not considered and the test was reported as a PASS.
This test passed correctly, it's Gramine which could not exit the child process cleanly (see #21) for details.
What's the issue here? Why would writing anything to stderr be considered an error? Debug logging is often done this way.
@boryspoplawski in my past experience you would want to check for unexpected messages in the stderr, they may not be impacting the workload (in certain cases) but still good to keep an eye out. We have discovered several issues with Gramine itself using this approach in our local CI. Even the IPC error, no matter how benign, it's still an issue with Gramine and something that is not caught by the opensource CI right now. We will continue to parse the stderr in our local CI. Michal asked me to file this bug, you can check with him and if it's not relevant then feel free to close the issue.
This test passed correctly, it's Gramine which could not exit the child process cleanly
Yes, I know this. I just wanted to update the bug with the latest status. Not asking to debug.
From my side: IMO it would be good to parse and catch such error messages in case they for some reason won't get reflected in the exit status. But this particular message would need to be somehow whitelisted then, as it's a known but benign bug.
