w3c
Verbose output of test builder isn't verbose enough
The create-all-tests script accepts --verbose and --validate flags, intended to help in debugging when a test plan won't build for whatever reason. Unfortunately, output only seems to be logged when:
- A part of the process completes successfully (such as parsing a CSV file); and
- explicit errors are thrown.
The script does not, however, log actions it is about to take, nor which file/row/column is being processed. As such, when an unexpected problem occurs, such as that highlighted in w3c/aria-at#1244, the test plan author is left at a complete loss about where to start looking for the issue.
Sticking with that example, there is literally no output when running node scripts\create-all-tests --verbose --validate --testplan=hang-sample:
- There are no successful actions for the script to report because it hangs almost immediately.
- There is no error thrown, because the regexp parser is technically doing its job albeit in a way that causes the script to hang.
As a result, it took me over an hour to track down exactly where the failure was occuring. Consider that if the script had output something like the following instead, it would've been a lot more obvious:
>node scripts\create-all-tests --verbose --validate --testplan=hang-sample
Validating row 0 in data/tests.csv for test plan "hang-sample".
Processing tests.csv column: testId
Processing tests.csv column: title
Processing tests.csv column: presentationNumber
Processing tests.csv column: setupScript
Processing tests.csv column: instructions
Processing tests.csv column: assertions
... it would've been immediately obvious that the script froze right after trying to validate the assertions column on that specific row of that specific file.
CC @howard-e
@ccanash
The seems like a pretty light-weight request with significant productivity benefits. I'm adding it to the ops and maintenance project as a P1.
@mcking65 this is noted. I will add it to triage and move it to the aria-at-app repo.
@ccanash This issue relates to scripts in the aria-at repo. The PR to fix it would presumably be submitted to that repo, so I don't think this issue belongs under the app.
This task had been added to the 2024 project by mistake, I have re added it to the 2025 maintenance project and will added it to our queue.
Hi @jscholes can you please test this and let us know if it works as expected? Thanks!