Create Github Action

[atifsyedali]

Would be great if we could use a Github Action instead of the app to push checks.

[SimenB]

Yes please! I'm playing around with using the built in problem matchers for tsc and eslint, and it works swimmingly: https://github.com/facebook/jest/pull/9223. Would be awesome to just stick an action in for test results as well 👍

[stefanbuck]

@SimenB are you aware of any existing Problem Matcher for Jest? The Jest output isn't really suitable for parsing using a RegExp or at least not that simple. I was thinking about writing a custom reporter to print a parser friendly output. How does this sound to you?

[SimenB]

The default jest output is certainly not suited. It should be a separate reporter (an idea might be that GitHub parses junit output?). Could probably reach out to GitHub about it - the js and ts templates they provide use Jest for testing so should be possible to work something out 🙂

[stefanbuck]

I proposed a similar idea using a json output and xpath https://github.com/actions/toolkit/issues/260.

Internal we use Danger.js to attach the jest output as a PR comment, but using a Problem Matcher would be a much nicer solution

DangerJS screenshot

[stefanbuck]

Problem matchers are for parsing console output and are run line by line. In you case you want a file and the runner won't when to process the file. One option would be to write an action to process the file and create annotations.

https://github.com/actions/toolkit/issues/260#issuecomment-572807059

Mhhh I'm undecided wether a GitHub Action or a jest reporter that produces a parsable output is the way to go. @SimenB what's your thoughts on this?

[SimenB]

There shouldn't be an issue to create a reporter that outputs something a problem matcher can parse. I think that's a good idea, as we could ship it by default in Jest and automatically activate it if we detect we're running in github actions? If not, I think GitHub should provide some other built in thing to support lint/test tools etc to just output a file in a given format to a given place, and have GH itself deal with reporting it to the user without having to write a full action

[stefanbuck]

I tend to prefer a new reporter mainly because it doesn't involve GitHub roadmap prioritization 😉 I'll look into this and keep you updated.

[stefanbuck]

I started working on this and the first issue I'm facing is the limitation to a single line for the message field https://github.com/actions/toolkit/issues/319. I guess there is not much we can do from our side to make the output useful in a single line!? Btw, should I raise a feature request in facebook/jest so we can continue the discussion there?

[SimenB]

@stefanbuck sorry, missed your last message. I still think a dedicated reporter makes more sense than a problem matcher. However, we could probably output some json on a single line per test failure or something - would that be parseable?

[stefanbuck]

To be honest I think we need both. A reporter which produces a single log line for each failing test and a problem matcher which is able to consume this log format. A problem matcher is just a really slick wrapper around the Checks API which integrates really well into the GitHub UI. What's the alternative to Checks/Problem matchers?

[mattallty]

@SimenB @atifsyedali @stefanbuck FWIW, I've been using another solution in https://github.com/mattallty/jest-github-action The way it works is:

• run jest with --testLocationInResults --json --outputFile=jest.results.json
• then parsing jest.results.json to get failed tests location as well as generating a markdown table of coverage
• Using the Check API to create annotations, and Issue Comments API to create a comment with coverage data

Seems to work great - I can't just get coverage data when some tests are failing.

[stefanbuck]

I do something similar with danger.js, but I really want to see this happening with problem matcher so I don't have to maintain another service.

Isn't the check API limited to GitHub Apps?

The Checks API functionality is available exclusively to your GitHub Apps. https://developer.github.com/v3/checks/

[hipstersmoothie]

Any GitHub action has access to an app token so they can make checks. Doesn't that work?

[mattallty]

I can confirm it's working

[stefanbuck]

Looks like the multiline issue I was facing can be solved by url encoding \n https://github.com/actions/toolkit/issues/193#issuecomment-605394935

[stefanbuck]

@SimenB I finally made it working

I ended up writing a custom reporter which produces an output which is follows a certain pattern. Please checkout this PR for details https://github.com/stefanbuck/jest-matcher/pull/2/files

How should we proceed with this? Do you think it's suitable to make this reporter part of the Jest core and active it whenever a GitHub Action env is detected?