adoptium
Consider adding new test material: jtreg-buffer
Assess and review the value of test material found at: https://github.com/rh-openjdk/jtreg-buffer.git
Criteria to consider, questions to answer:
- what is the purpose of the new test material
- does it fill a gap in AQAvit testing (or does it duplicate other tests already present)
- is there good documentation to point to
- are test dependencies and prereqs known/tracked (including version levels)
- is the test source repo stable and well-maintained
- can we pin to a particular tag or SHA to help shield us from test repo changes that may break users of AQAvit
Assess and review the value of test material found at: https://github.com/rh-openjdk/jtreg-buffer.git
Criteria to consider, questions to answer:
* what is the purpose of the new test material
The repository contains jdk targeting tests, wich did not (yet) fit into jdk's regression suite. see; https://github.com/judovana/aqa-tests/blob/9c40ebbf7c349d228c84b8df2d10a5c238cfbeb9/external/jtreg-buffer/README.md#external-jtreg-buffer-tests and https://github.com/rh-openjdk/jtreg-buffer/blob/main/README.md
* does it fill a gap in AQAvit testing (or does it duplicate other tests already present)
It is not duplicating anything, but afaik is not filling any gap. It is targeting weird corner cases of jdk, not (yet) suitable for upstream regression suite. It can be used as temporary home of tests, waiting for upstreaming to JDK. Or harbour tests, targeting jdk, but unfit for jdk contribution for any reason
Thats why I had choose external as target group, but I have no issues to put it to any other recomended group.
* is there good documentation to point to
Sure! ..as far as I can judge...
- it is jtreg based
- contanst click and run top elvel script
- all custom parts are in readm.md
* are test dependencies and prereqs known/tracked (including version levels)
The suite pulls jtreg.tar.gz as it needs - one for 8, second for others. It can run with any local jtreg tarball
* is the test source repo stable and well-maintained
Yes.
* can we pin to a particular tag or SHA to help shield us from test repo changes that may break users of AQAvit
Yes. After last issue I had tagged all rh-openjdk testsuites. So the latest tag can be used. But that brought up an idea, that the versin/hash/tag/branch which will be set in aqavit as stable, should be easily overwrite-able from command line, so I (or anybody interested) may run custom version of the suite. Intentionally, or just to try.
Especially with this one - when new test is added, it is added with purpose, and I would like to imediately be able to run it - without modifying code, and without disturbing the global stability by making it globally available immediately.
Based on this ticket, it was written in https://github.com/adoptium/aqa-tests/pull/5167#issuecomment-2037807288 that openjdk/dev should be used.
IIUC, it is https://github.com/adoptium/aqa-tests/blob/master/openjdk/playlist.xml , and where dev is definitly correct level, I'm, ot usre if that group openjdk is correct. Two reasons
- that one is cloning always upstream repo, where I still ened to clone the buffer repo
- that one is targetting topic-based parts of usptream jtregs, where this one of mine is a bit of all
Thats why I have seelcted external group as target.
In addition, all openjdk tests are targetted to some functionlaity, like math, graphic, crypto... But this jtreg-buffer is testing generally everyting.
Actually one vote against external is that it do not need any deps, nor anything from exxternal . So maybe system or functional may be the best?
Currently category openjdk has only one build.xml and playlist.xml , which is different from other categories ( tiered down tests with sub directory, each directory has its own playlist.xml and build.xml).
To harbour those tests in openjdk category as temporary home before upstreaming to JDK we can
- update build.xml and playlist.xml - which incorporates downstream tests with upstream tests. Feels simple, but make test structure complex
- tier down the structure of openjdk to be similar to other category - both downstream and upstream tests have its own build.xml ( clone the buffer repo) and playlist.xml. Common function moves to root common build.xml. A bigger change, but more clear and straightforward. Also it would be easy if we need to add other tests later.
I'm also thinking instead of making incubating tests as dev level would it be better to add a dev category? So all incubating tests locate in same locations and run under same category. The cons is when it's ready to move out of incubation might need to do some migration.
Currently category openjdk has only one build.xml and playlist.xml , which is different from other categories ( tiered down tests with sub directory, each directory has its own playlist.xml and build.xml).
To harbour those tests in openjdk category as temporary home before upstreaming to JDK we can
I would not be thinking baout it in that way. I would think about it simply as of new testsuite. Some of the tests aren ot upstream-able.
*update build.xml and playlist.xml - which incorporates downstream tests with upstream tests. Feels simple, but make test structure complex
I would vote -1 for that. When I first time saw the openjdk/playlilst.xml I got frozen in fear and astonishment.
*tier down the structure of openjdk to be similar to other category - both downstream and upstream tests have its own build.xml ( clone the buffer repo) and playlist.xml. Common function moves to root common build.xml. A bigger change, but more clear and straightforward. Also it would be easy if we need to add other tests later.
This sounds correct. I'm not sure if I'm experienced enough to do that. Will that "only" mena that instead of :
openjdk
|---playlist.xml
there will be
openjdk
|---math
| |--playlist.xml (with extracted the math targets from original playlist.xml)
|---something else
| |--playlist.xml (with extracted thesomething else targets from original playlist.xml)
...
|---some custom
| |--playlist.xml (with a bit cusotm jtregs, like eg the jtreg-buffer)
...
?
Will the results be backward compatible?
I'm also thinking instead of making incubating tests as dev level would it be better to add a dev category? So all incubating tests locate in same locations and run under same category. The cons is when it's ready to move out of incubation might need to do some migration.
I do not understend the wider picture enough to have opinion here.
If the bigger refactoring of openjdk/playlist.xml will happen, can the jtreg-buffer reside in functional or system in meantime? (aka the most similar playlyst.xml structure to future refactored openjdk/playlist)
Thanx!
Mark as closed since https://github.com/adoptium/aqa-tests/pull/5308#issuecomment-2245581937 these tests will be run out of a vendor specific pipeline