epiphany icon indicating copy to clipboard operation
epiphany copied to clipboard

[FEATURE REQUEST] [epicli test] Print total number of failed tests

Open to-bar opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. epicli test may produce a big number of xml files (in spec_tests sub-directory) and there is no information about total failed tests. One has to check all xml files manually, for example using grep:

grep -Por 'failures="\d+"'  /workspaces/epiphany/clusters/build/alma-test-big/spec_tests | grep -cv '"0"$'
1

Describe the solution you'd like Print aggregated test results, for example: Total tests: 64 [failures: 1, errors: 0, skipped: 0]

Describe alternatives you've considered Configure rake to create single (common) output file for all groups and hosts.

Additional context n/a


DoD checklist

  • Changelog
    • [ ] updated
    • [ ] not needed
  • COMPONENTS.md
    • [ ] updated
    • [ ] not needed
  • Schema
    • [ ] updated
    • [ ] not needed
  • Backport tasks
    • [ ] created
    • [ ] not needed
  • Documentation
    • [ ] added
    • [ ] updated
    • [ ] not needed
  • [ ] Feature has automated tests
  • [ ] Automated tests passed (QA pipelines)
    • [ ] apply
    • [ ] upgrade
    • [ ] backup/restore
  • [ ] Idempotency tested
  • [ ] All conversations in PR resolved
  • [ ] Solution meets requirements and is done according to design doc
  • [ ] Usage compliant with license

to-bar avatar Mar 23 '22 16:03 to-bar

Would be nice to consider another thing related to test results:

  • when running tests against the cluster that has configured components that use role mapping for filebeat under configuration/feature-mapping , but doesn't include logging component, it will generate empty spec_tests result file for filebeat
  • it happens because under tests/spec/Rakefile file we have specified to run tests against all groups in inventory, but in tests/spec/spec/filebeat/filebeat_spec.rb we are skipping tests if logging component is not enabled
  • empty files are read by Azure to publish test results which lead to warnings about tests for filebeat

Thanks to @przemyslavic for explanation.

If you think that it requires separate ticket, please comment.

cicharka avatar Apr 20 '22 13:04 cicharka