testomatio
Failures count in RunGroup sub-group statistics does not match realistic strategy aggregation
Describe the bug When a RunGroup is created with the realistic merge strategy,
- the number of failed tests displayed in the sub-group statistics does not match the actual number of failures recorded in the test runs;
- all test cases are unique across different runs (unique titles and Testomat IDs); Despite this, the aggregated statistics display only 12 failed tests, while the actual runs contain more failures (e.g., 13 failed in one run alone, plus additional failures in others).
Precondition:
- RunGroup is created with the realistic merge strategy
- Unique test cases are created
To Reproduce
- Launch multiple runs inside the RunGroup, each with unique test cases
- Execute tests and mark statuses according to the results
- Finish all launched runs
- Open the RunGroup view in the sidebar
- Compare the number of failed tests in the per-run column vs. the sub-group aggregated statistics
Expected behavior
- For RunGroups with realistic strategy, the system should always prefer results from the last executed run (excluding skipped tests)
- Since all test cases are unique across runs, the aggregated failed count should equal the total number of failures from the latest run results
Screenshots
Desktop (please complete the following information):
- Application: production
In general, there's definitely at least one FLACK, and there's no talk of any unique tests here. However, it seems our main problem is that when test_id=NULL, the counters don't work. This seems very bad, especially for automated tests, which fail without creating an automated test, meaning with test_id=NULL.
I suggest grouping by test_title in the case of test_id=NULL. This is entirely possible.
There can be new tests that were not imported before reporting, so test IDs are missing, and tests are not included in RunGroup counters.
