aqa-test-tools
aqa-test-tools copied to clipboard
Optimize Parsing & Storage of Raw Values
Problem Description
In order to extract Perf results for builds that have multiple tests in them (https://github.com/AdoptOpenJDK/openjdk-test-tools/issues/136), we decided to store raw values in the aggregate info data structure.
The design is discussed: https://github.com/AdoptOpenJDK/openjdk-test-tools/issues/136#issuecomment-582062349
Pros of current design:
- Makes parser simpler
- Needs to done only once when the build finishes
- Different front end tools doesn't require special parsing code.
- Avoids somewhat duplication of code in the front end since front end tools would require similar logic as the one that's already there in the back end.
Cons of current design:
- Duplicates raw values by collecting it from different places/builds and putting them in an array
We could re-visit this design to see whether we can make this more efficient to possibly avoid duplicating raw values.
Related PR: https://github.com/AdoptOpenJDK/openjdk-test-tools/pull/179
As requested:
We mainly have 2 scenarios for running perf tests. Here's a short summary:
| Scenario | Description | Uses Aggregate Data | Example |
|---|---|---|---|
| Using Openjdk-tests Framework (TKG) | For each JDK, we run multiple tests multiple times in one Jenkins Build. We do the same for the baseline JDK in another pipeline. | Yes | Test_perf/job/Test_openjdk11_j9_sanity.perf_ppc64le_linux_siena/ ITERATIONS=4 BUILD_LIST=perf/quarkus TARGET=sanity.perf |
| PerfNext | It uses 3 levels: Grandparent, Parent, and Child. For every test, one grandparent job is launched. If interleave is selected, then 2 parent builds are launched by grandparent. Parent build launches multiple child builds, depending on the number of iterations for that specific test. We monitor the parent build. | Yes | Test_perf/job/PerfNext-Parent |
All perf tools (Tabular View, Perf Compare, & Perf Graphs) use aggregate data structure as well.
Please let me know if you need more clarification on something.