skuba
skuba copied to clipboard
GitHub test annotations can fail validation
I observed the following on a couple flaky database tests:
Summary of all failing tests
FAIL src/app.int.test.ts
● Test suite failed to run
Connection terminated
at Connection.<anonymous> (node_modules/pg/lib/client.js:132:36)
at Socket.<anonymous> (node_modules/pg/lib/connection.js:57:12)
Test Suites: 1 failed, 23 passed, 24 total
Tests: 182 passed, 182 total
Snapshots: 61 passed, 61 total
Time: 56.839 s
Ran all test suites.
Failed to annotate results.
HttpError: Validation Failed: {"resource":"CheckRun","code":"invalid","field":"annotations"}
Unfortunately the error isn't very descriptive and I can't see anything obvious that would trip up GitHub's validation. I tried to fake a similar error locally and it seemed to produce a reasonable list of annotations:
[
{
annotation_level: 'failure',
path: 'src/app.int.test.ts',
start_line: 1,
end_line: 1,
message: 'Connection terminated',
title: 'Jest'
}
]
Should we log the parameters we call the checkRun with if we fail? I suspect the only way will be a bit of trial and error against the checkrun API to see what it doesn't like.