aqa-tests Run all external

This is script designed to verify/test/fix all external tests on local-host. It is designed to help with: https://github.com/adoptium/aqa-tests/issues/5575#issuecomment-2341256974

It also includes results from the January run

Whether to integrate it at the end is disputable.

Dec 03 '24 15:12 judovana

@adamfarley @sxa hi! Afaic this PR is not for merge, but I had finally composed the matrix of pass/fail for external tests:

https://github.com/judovana/aqa-tests/tree/runAllExternal/external#passfail-matrix-112025

I think I wills tart to fix it one by one... It will be very slow run :) Any other ideas what next? Maybe we should merge only the table? IDK....

Mar 11 '25 17:03 judovana

@adamfarley @sxa hi! Afaic this PR is not for merge, but I had finally composed the matrix of pass/fail for external tests:

https://github.com/judovana/aqa-tests/tree/runAllExternal/external#passfail-matrix-112025

I think I wills tart to fix it one by one... It will be very slow run :) Any other ideas what next? Maybe we should merge only the table? IDK....

Good work @judovana!

I suggest merging the table, excluding the failing external tests (so it runs green, or at least yellow), and fixing the individual tests in separate PRs to enable maximum coverage in the short/medium term.

What do you think?

Mar 12 '25 11:03 adamfarley

The table is generated by java class from branch, which crawls data in results pushed in that branch and links those data.. the data structure is dictated by the script in that branch. So pushing the table as it is is not exactly straightforward :)
Note, that I do not want the current run-all and genertate-table merged as it is - I actually discourage it. So maybe just link the table to the fork+branch? Maybe to the generic branch in adoptium aqa-tests, so other can contribute if anything is fixed? Or maybe rework the run_all.sh and the results so it is merge-able (+and adjust generateTable accordingly)? tbh, IDK :(

I'm not fan of excluding tests without proper investigations, so I would rather keep state as it is, but to fix it all, is a lot of work. Even to gather results is a lot of work. On contrary I belive that once I fix soem first ten, it will become faster.

So maybe really, contribute the table as it is, and then big fixing/excluding can start, and somwhen in future, the table will be regenerted. I do not have strong opinion on this, except my determination to make it much more green:)

Mar 12 '25 13:03 judovana

We will not store test results in this repository.

So there will be no plan to merge this PR here. For such a change, I would have expected a design discussion or at least an issue outlining the direction and plan, to have saved you the time on this draft. If we are to utilize the work you have done, we will have to move the reporting aspect under a new/separate repository. Before that, an overall outline of the design and plan would be good.

I appreciate the effort you have gone to, but would like to have discussed direction first (as per the initial comment on the topic, https://github.com/adoptium/aqa-tests/issues/5575#issuecomment-2341216147)

Mar 12 '25 14:03 smlambert

We will not store test results in this repository.

I agree. And I was clear on it in previous comment, wasn't I? Please?

As for publishing the table itself is different topic. It have sense, because from time to time I hear people running it and it is having failures, so asking what is the expected pass/fail matrix. But I do not insist, really. Afaict it is perfectly ok if it is in fork, or in other branch.

Also we already had this discussion? https://github.com/adoptium/aqa-tests/issues/5575

I would like to make the table as green as possible, which is subject of another discussion, because it seems there is other work on external tests in progress, and I do not want to muddle below anybody's hands. I have a topic to discuss it with adoptium for half a year, and those results were laying on my machine for two months, and to fix the individual test, I needed the matrix, and htmls/github is good place. And if it is useful, then why not to share it.

I know you are bussy with other stuff, and so am I, so it is rolling extremely slowly. Still I have great interest in external tests, so I found few cycles to at least compose the matrix for wider audience.

Mar 12 '25 15:03 judovana

I agree. And I was clear on it in previous comment, wasn't I? Please?

My message (re: no test results) was for Adam's awareness

Mar 12 '25 17:03 smlambert