w3c
Adding report run to test queue incorrectly removes report from report page
I ran into this in sandbox while trying to test NVDA automation. This bug prevents making a report for a new version of an AT by forcing removal of any already final report for the same AT/browser combination. It is critical that we don't pull back a published report. We need to make new reports for the same AT/browser combination. That is the primary use case for automation.
To reproduce this bug:
- Identify a AT/browser/test plan combination with a report on the reports page.
- Add a run to the test queue for that test plan and AT/browser combination.
Expected result: A new empty test run is added to the test queue. I was expecting to run it with automation and have the responses compared to the previous final report.
Actual result:
- The report is removed from the report site.
- The test queue has previous results for that AT/browser combo and a "Mark as Final" button.
It appears this bug is due to changes described by by @alflennik in #796. We may need to find a way to roll back some of those changes ASAP.
Here are some general reporting requirements that I thought were already understood based on all the automation discussion. However, it seems we don't yet have alignment on these.
- Keep all reports that have been marked final.
- If a report meets the following criteria, surface it on the reports page:
- It has been marked final.
- It is generated from a test plan version that is recommended, OR the test plan version is in candidate review and that test plan does not have a recommended version.
- It includes data for the latest AT version and latest browser version that have been tested for a given AT/browser combo
- Reports for older AT and browser versions should be maintained in the system so we can use them when we add trend reporting.
- When a report run for a given test plan version is added to the test queue, it should always be empty; do not pull back an existing report from the report site. We do not have a requirement to edit reports; we will just replace them but only when specific conditions are met.
- When a report is marked final in the test queue, it is published to the reports page if it meets the above conditions. Otherwise, e.g., it is for older versions of AT or browser, it is kept in the system for trend reporting. If it is for the exact same version combination as a report previously marked as final, warn the admin that it will replace the current report.
- If a report is not available on the reports page because the test plan version is draft, is candidate but has a recommended version, or is not for the latest AT/browser version, the report should be accessible from the test plan versions page for that test plan and from the report status dialog on the data management page.
Currently, we do not specify the AT/browser version that a tester will use when completing a report run. We allow the tester to use any version, and we track which versions are used. That is still good for draft and candidate plans. As described in #809, that is not adequate for recommended test plans. We will address that in Q1. However, in the meantime, we need to keep all final reports, not pull back existing reports and make reports that are not available from the reports page available as described in requirement 5. Requirement 5 was previously described in the requirements for the test plan versions page and report status dialog.
Hi @mcking65, thanks for clearly laying out the situation and your expecations. We discussed this issue as a team and brainstormed some solutions. The hard part is that this is not a bug that we can quickly roll back - the logic of pulling reports back to the queue has been the longstanding behavior of the app for a few years now.
Before we fully address this issue we felt that a quick action we could take is to show an error dialog in the case that the report already exists.
cc @lolaodelola @ccanash
Hi @mcking65 I created the task for the dialogue here #898 which is in progress
