EPIC: Centralize and update test documentation

[smlambert]

Since test doc lives all over the place it is hard to find and navigate, we will do a reorg/rewrite based on feedback received through slack, etc.

Based on comments captured below in this issue, it will be tackled in parts

• Tutorials / Learning-oriented (longer learning session)
  • [ ] Add new platform support
    • reference: https://github.com/adoptium/aqa-tests/wiki/Adding-AQAvit-support-for-a-New-Platform
  • [ ] Incorporate vendor tests
    • example: See how smoke tests are incorporated using vendor test repos from the build pipeline code
    • provide another simple example of laying down test material in a repository separate from the main ones that are directly part of "AQAvit", and the minimum requirements for what is currently needed to execute those tests (playlist.xml, build.xml files and testcases)
  • [ ] Run tests in a Github action
  • [ ] Use TRSS to triage a release
  • [ ] ... other tutorials on commonly conducted activities
• How-to Guides / Task-oriented (quick lessons)
  • [ ] How to verify a release (with testenv.properties) (see also https://adoptium.net/docs/aqavit-verification/)
  • [ ] How to match the right aqa-tests release branch with a particular JDK
  • [ ] Rerun a failing test in a Grinder job
    • reference: https://github.com/adoptium/aqa-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins#rerun-a-failing-openjdk-regression-test to show how to rerun a failing testcase (different from rerunning a failing test target)
    • Also show the case for rerunning a failing test target
  • [ ] ... other quick how-to's based on lightning talks
• Reference / Information-oriented (detailed encyclopedic knowledge)
  • [ ] TKG features reference
  • [ ] Jenkins features reference - Can start with https://github.com/adoptium/aqa-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins#reference
  • [ ] TRSS features reference
  • [ ] Terminology reference (top-level targets down to individual test cases)
  • [ ] Jenkins label schema (refer to smlambert branch addressing https://github.com/adoptium/infrastructure/issues/93)
• Explanation / Understanding-oriented ('why' things are designed a certain way, 'why' AQAvit exists)
  • [x] AQA Definition (Manifesto) - https://github.com/adoptium/aqa-tests/pull/4419
  • [x] Scope of AQAvit (AQAvit verification, Cloud compatibility, Developer support)
  • [x] Why the '3 layer cake' architecture (via https://github.com/adoptium/aqa-tests/pull/4686)

Noting that this EPIC specifically targets the content that needs to be created, updated and centralized. There will be a separate issue to cover how to display the final set of docs (with some prototyping happening under https://github.com/smlambert/smlambert.github.io and viewable at https://smlambert.github.io/

[smlambert]

Test-related documentation (some of which is not relevant to most consumers of AQA, but listed for completeness:

https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/buildenv/docker/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/doc/Triage.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/doc/TriageChecklist.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/doc/userGuide.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/external/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/functional/SyntheticGCWorkload/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/jck/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/openjdk/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/perf/README.md https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/system/README.md https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/README.md https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/PerfNext/README.md https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/SmartMedia/README.md https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/TestResultSummaryService/README.md https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/master/test-result-summary-client/README.md https://github.com/AdoptOpenJDK/TKG/blob/master/README.md https://github.com/AdoptOpenJDK/stf/blob/master/README.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.build/docs/build.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.core/docs/STF-GettingStarted.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.core/docs/STF-Internals.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.core/docs/STF-Manual.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.load/docs/README.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.load/docs/arguments.md https://github.com/AdoptOpenJDK/stf/blob/master/stf.samples/docs/readme.md

https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/master/README.md (and all its subdirs) https://github.com/eclipse/openj9-systemtest/blob/master/README.md and https://github.com/eclipse/openj9/blob/master/test/README.md https://github.com/eclipse/openj9/tree/master/test/docs and https://github.com/AdoptOpenJDK/openjdk-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins https://github.com/AdoptOpenJDK/openjdk-tests/wiki/Guidance-for-Creating-OpenJDK-Test-Defects https://github.com/AdoptOpenJDK/openjdk-tests/wiki/Graduated-Testing-&-Test-Numbers

and probably some of the most used / useful guides: https://github.com/eclipse/openj9/wiki/Reproducing-Test-Failures-Locally

[smlambert]

Use https://diataxis.fr/ as a guidance for this work.

• Tutorials / Learning-oriented
• How-to Guides / Task-oriented
• Reference / Information-oriented
• Explanation / Understanding-oriented

Hopefully within the next weeks, I will be able to outline some content and categorize it according to the above areas

[smlambert]

Do not forget to add a section for the labeling schema we use (and also so that as it is extended, it continues to make sense and is not extended in non-intuitive ways), original notes here: https://github.com/smlambert/openjdk-infrastructure/blob/labels/docs/jenkinslabels.md

[smlambert]

C4Dynamic
    title The 3 layers of AQAvit 

Boundary(b, "AQAvit") {

System_Boundary(b2, "Monitoring, Aggregation & Reporting") {
    Component(c4, "TRSS", "Test Results Summary Service", "Stores and represents aggregate test results, allows for advanced search and reporting capabilities.")
    }

    System_Boundary(b1, "CI Layer") {
      Component(c3, "Jenkins", "CI Server", "Multiplex/parallelize test jobs across multiple nodes and platforms.")
    }


    Container_Boundary(b01, "Test Nodes") {
      Component(c01, "TKG", "Test Kit Gen", "Provides base level functionality for standardizing the many disparate test frameworks.")

   }


}

    BiRel(c4, c3, "TRSS polls CI for console output", "Jenkins API")
    BiRel(c01, c3, "Jenkins in contact with Jenkins agent on test node")
   
    UpdateRelStyle(c01, c3, $textColor="red", $offsetY="-20", $offsetX="10") 
    UpdateRelStyle(c4, c3, $textColor="red", $offsetY="-40", $offsetX="10")
    
   UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")

[smlambert]

[smlambert]

[smlambert]

Other docs to include:

• AQA definition to be updated from AdoptOpenJDK Quality Assurance v1 to Adoptium Quality Assurance Definition
• Etiquette for handling PRs (if contributor asks for a review from a particular reviewer because they know the code well, we should respect that request, within reason. Also unless there is something breaking the world, it is better to take time with reviews, it is not a race)
• Additional Guidance for code reviewers (how to determine what additional testing is required, its is not all automated PR testing, by choice and for good reason)
• Terminology reference (what does 'test target', 'dynamic agent', 'dynamic compilation', 'dynamic parallelization', 'TAP', etc refer to?)
• Scope of AQAvit, various ways it can and should be used:
  • AQAvit verification (9 top-level targets currently forming the definition of that)
  • Cloud Compatibility Program (9 top-level targets + 1 more, still being defined)
  • Development support (all top-level targets in the grid, including any and all dev level testing)
  • Vendor-specific proprietary test support (and and all top-level targets in the grid, plus overlaid vendor tests)
• Jenkins label schema (including all existing/known labels in use, by the project and downstream/vendor use)

[Ndacyayisenga-droid]

Hi @smlambert. Can we work on this together?

[smlambert]

Hi @smlambert. Can we work on this together?

Happily @Ndacyayisenga-droid !

[llxia]

It would be nice to have docs about the AQAvit release and add the usage of testenv.properties in the AQA repo (https://adoptium.net/docs/aqavit-verification/ ).

[smlambert]

re: https://github.com/adoptium/aqa-tests/issues/1558#issuecomment-1715661898 - agree, added it to the description above (in the "how-to" section).