backdrop
Upload Simpletest Results as an Artifact in GHA
Description of the need
Debugging test failures in GitHub Actions is often difficult, because you can't see what is happening like you can when running tests locally through the UI. In particular, the contents of HTML pages that are loaded (and verbose messages) are helpful in that it shows you what the HTML browser sees while executing the test.
Proposed solution
We could make things more transparent as to what's happening in our GitHub Action test runs by uploading the verbose output as an artifact and attaching it to the test runs.
Alternatives that have been considered
I usually run tests locally when there is a failure, but occasionally there is an error that happens only on GitHub (like we've been experiencing with #2277).
Additional information
This can be achieved by adding this to the bottom of our test YML files:
- name: Save Test Results
uses: actions/upload-artifact@v4
if: always()
with:
name: simpletest-output
path: files/simpletest
FWIW I added these lines to the GHA test for backup_migrate and they worked great. Thanks!
PR filed at https://github.com/backdrop/backdrop/pull/4730 that enables saving the SimpleTest output as artifacts in both the regular and database-config tests.
To test:
- Open the PR.
- Click on any test run for simpletest.
- Expand the test result section labeled "Save Test Results"
- Click the link inside this section, as shown below:
- Open the downloaded archive and verify it contains test run contents. It will be the HTML output of every test that was run.
Code looks good to me 👍🏼 ...if I can make a suggestion, that would be to add the PR number as a prefix to the artifact files + a timestamp as a suffix. Otherwise, assuming that people will be downloading these locally to extract and investigate/troubleshoot, it would be annoying to have the operating system naming the downloaded files with (1) (2) or whatever suffix, and then having to figure out which one is for which PR and if it was before or after a specific commit.
Other than that, lets please get this in! 🙂
Finding the pull request number is apparently difficult, because tests can run on things other than just pull requests. Big long thread on that here: https://github.com/actions/checkout/issues/58
...from a quick research, I believe that ${{ github.event.number }} should work for the PR number.
...and https://stackoverflow.com/questions/60942067/get-current-date-and-time-in-github-workflows has an example on how to get the date from a GHA step.
But I don't want to derail this. I'm happy to get it in as is.
Thanks, yeah I started looking into it and just realized it's messier than you'd think. The current approach works and (hopefully) downloading artifacts will be a rare need; usually only when we're dealing with GHA-specific issues.
I merged https://github.com/backdrop/backdrop/pull/4730 into 1.x only, since the testing framework was reworked quite a bit in https://github.com/backdrop/backdrop-issues/issues/2277, which is 1.28.0 and higher.
...realized it's messier than you'd think.
I don't know about that. Seems pretty simple to me, and it works: https://github.com/backdrop/backdrop/pull/4731
Feel free to close the follow-up PR if you don't agree.
With how rare (hopefully) downloading simpletest results is; this only affecting developers; only certain browser situations; and the addition of event number potentially causing problems when running tests outside of PRs, I think I'd like to be able to just close this for 1.28.0. Thank you for filing it and we can revisit if we find it to be actually problematic.