Support skip and broken results in LTP parser

[acerv]

The current runltp-ng JSON parser is not taking into account broken/skipped tests, which are generated by LTP tests results. In particular TSKIP/TCONF and TBROK.

[asmorodskyi]

TCONF/TSKIP was in LTP for ages , but initially it was ignored . I want to double check maybe we missing some idea why it was done like that ? Maybe there some pitfall in also taking into account TCONF/TSKIP ?

@richiejp , @pevik , @foursixnine WDYT ?

[acerv]

@asmorodskyi the only usage of OpenQA::Parser::Format::LTP is inside tests/publiccloud/run_ltp.pm implementation. The tests/kernel/run_ltp.pm doesn't use JSON file because LTP logs are parsed instead, as you can see from the following implementation: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/d813b90a08e63eadbf28a622d197a3a2a1e2cc9f/tests/kernel/run_ltp.pm#L211

With this modification we are aligning tests/kernel/run_ltp.pm and tests/publiccloud/run_ltp.pm to the same results in openQA.

[pevik]

@richiejp , @pevik , @foursixnine WDYT ?

@asmorodskyi Did Andrea answer your question?

I'm not familiar with tests/publiccloud/run_ltp.pm, but I changed tests/kernel/run_ltp.pm in https://github.com/os-autoinst/os-autoinst-distri-opensuse/commit/a760f7d91721ac159a1c2ffdf0b7fbb40320fb8e to handle TCONF as skipped, i.e. gray instead of green and blue border around box, see: https://openqa.opensuse.org/tests/2763684. Public cloud tests does not have it: https://openqa.suse.de/tests/9615085#step/cacheflush01/1. I'd just recommend to follow this approach for better readability (sometimes changing from TPASS to TCONF shows a problem, i.e. missing module which should be available. But that's a separate problem.

[asmorodskyi]

@asmorodskyi the only usage of OpenQA::Parser::Format::LTP is inside tests/publiccloud/run_ltp.pm implementation. The tests/kernel/run_ltp.pm doesn't use JSON file because LTP logs are parsed instead, as you can see from the following implementation: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/d813b90a08e63eadbf28a622d197a3a2a1e2cc9f/tests/kernel/run_ltp.pm#L211

With this modification we are aligning tests/kernel/run_ltp.pm and tests/publiccloud/run_ltp.pm to the same results in openQA.

ok I see . I can not say that they truly aligned because we still have same logic in two different places ... but it certainly move into right direction :+1:

[asmorodskyi]

@richiejp , @pevik , @foursixnine WDYT ?

@asmorodskyi Did Andrea answer your question?

I think yes

[acerv]

@asmorodskyi thanks!

[acerv]

@m-dati thanks for your suggestion. Please check if code is ok. I added details update in order to have a proper External results page in openQA.

[m-dati]

@m-dati thanks for your suggestion. Please check if code is ok. I added details update in order to have a proper External results page in openQA.

Thanks to you too, I tested locally those new changes that worked fine on my openqa localhost web gui, including last runltp timeout update that resulted to pass, pls see here attached the exported test data: tst486_PYTH-UP2.zip

[acerv]

@m-dati I seen the results and I can consider this task done from my side. @kalikiana Can we merge this patch?

[Martchus]

The change will be merged automatically when tests are passing. Maybe you just need to rebase against master to make them work again.

[acerv]

@okurz done

[okurz]

So the one check failing is "codecov/patch — 33.33% of diff hit (target 100.00%) " meaning that unit test coverage for the changed code sections should be added. @acerv can you look into extending tests accordingly in t/30-test_parser.t?

[acerv]

@okurz if you take a look at the latest modifications, I have added the coverage for that part of the code. I'm stucking with tidy atm, but I'll handle it (hopefully)