[JENKINS-7781] "Test Statistics Grid" double counting aggregated tests

[TobiX]

I'm trying to use the dashboard-view "Test Statistics Grid" portlet, and I've noticed that it is double counting tests where the jobs use "Aggregate downstream test results". I have one main job with three downstream jobs. The main job has a few tests, but the majority of the tests are in the three downstream jobs. Presumably because Hudson is aggregating the test results, I am seeing double counts in the portlet. I would have expected it to count any given test only once.

In actuality, my main job has 4 tests, and the three sub jobs have 684 tests in total. dashboard-view is reporting that the main job has 684 tests (so it is counting the subs, but not its own), in addition to the sub jobs, for a total of 1368 tests.

It seems Hudson itself does not report the 4 tests in main in the Aggregated Test Result, so perhaps missing those is not the fault of dashboard-view. However, I should note, the "Test Trend Chart" portlet gives the correct numbers (684 without the main included, 688 with).

Please let me know if anything should be tried or you need more information.

I've pasted the table generated below, although I fear it will be difficult to read.

Test Statistics Grid
Job Success Failed Skipped Total

1. % # % # % #
   SuccessTest Result: 0 tests in total.mainjob 684 100% 0 0% 0 0% 684
   SuccessTest Result: 0 tests failing out of a total of 26 tests.subjob1 26 100% 0 0% 0 0% 26
   SuccessTest Result: 0 tests failing out of a total of 563 tests.subjob2 563 100% 0 0% 0 0% 563
   SuccessTest Result: 0 tests failing out of a total of 95 tests.subjob3 95 100% 0 0% 0 0% 95
   Total 1368 100% 0 0% 0 0% 1368

---

Originally reported by ellisonch, imported from: "Test Statistics Grid" double counting aggregated tests

• status: Reopened
• priority: Minor
• resolution: Unresolved
• imported: 2022-10-30

[TobiX]

mambu:

Is this still an issue?

It seems that the dashboard is behaving correctly anyway because you said:

In actuality, my main job has 4 tests, and the three sub jobs have 684 tests in total. dashboard-view is reporting that the main job has 684 tests (so it is counting the subs, but not its own), in addition to the sub jobs, for a total of 1368 tests.

So if jenkins is not reporting the 4 tests in the main job either, then I don't think we are doing anything wrong in the dashboard portlet.

[TobiX]

mambu:

I'm closing this one. Reopen it, if still an issue.

[TobiX]

ellisonch:

It still seems like an issue to me. I will try to attach more information.

[TobiX]

ellisonch:

I've added an attachment showing what I mean. If you add up all the child tests except for the csmith-tests, you get 1554. This number is being reported for c-semantics as well. So these tests are double counted in the 3669 total (which makes the % success total meaningless). Furthermore, the 5 tests that belong to c-semantics itself are not counted at all in these numbers. Finally, the csmith-tests are not being double counted, perhaps because the job is disabled. I don't really understand any of these behaviors.

Let me know if you need more info.

[TobiX]

mambu:

I don't understand the behaviour either, but you said that this also happens when displaying the test results in jenkins. This is important because we don't do any special processing with the test results and we get them directly from jenkins APIs.
Can you please show me also the standard test result for the main job? Does it show the 5 missing tests?
Need to check about the disabled job: do you think we should include it as well in the total count? If not we could "mark it" in a special way so that it does not cause confusion in the table?
What version of the dashboard-view plugin are you using?

[TobiX]

ellisonch:

On the c-semantics project page, it says "Latest Test Result (85 failures / ±0)" without double counting that. If I click that link, it gives me "5 tests", which is correct for the main job. If I click to "aggregated test result" on the left, I see good numbers. 85 failures, 1554 total tests, 526 unit tests, 33 integration, 777 torture, 127 third party, 91 negative, and N/A csmith. It does leave out the 5 tests in the main job, but it doesn't double count anything. It's been so long since my initial report that I'm not sure what I used to see.

We're using Jenkins 1.444 and Dashboard View 2.1 now.

I won't be able to upgrade or try installing different things because I'm leaving the group. I do know that the double counting has happened pretty reliably over the past few years. Hope this helps!