skuba icon indicating copy to clipboard operation
skuba copied to clipboard

GitHub test annotations can fail validation

Open 72636c opened this issue 2 years ago • 1 comments

I observed the following on a couple flaky database tests:

Summary of all failing tests
 FAIL  src/app.int.test.ts


  ● Test suite failed to run

    Connection terminated

      at Connection.<anonymous> (node_modules/pg/lib/client.js:132:36)
      at Socket.<anonymous> (node_modules/pg/lib/connection.js:57:12)


Test Suites: 1 failed, 23 passed, 24 total
Tests:       182 passed, 182 total
Snapshots:   61 passed, 61 total
Time:        56.839 s
Ran all test suites.
Failed to annotate results.
HttpError: Validation Failed: {"resource":"CheckRun","code":"invalid","field":"annotations"}

Unfortunately the error isn't very descriptive and I can't see anything obvious that would trip up GitHub's validation. I tried to fake a similar error locally and it seemed to produce a reasonable list of annotations:

[
  {
    annotation_level: 'failure',
    path: 'src/app.int.test.ts',
    start_line: 1,
    end_line: 1,
    message: 'Connection terminated',
    title: 'Jest'
  }
]

72636c avatar Apr 07 '22 22:04 72636c

Should we log the parameters we call the checkRun with if we fail? I suspect the only way will be a bit of trial and error against the checkrun API to see what it doesn't like.

samchungy avatar Apr 10 '22 03:04 samchungy