Use the option -f github to get the typical V2.checks formatter but with github actions ::error commands written out after each failing test. Run this in a github action and it will annotate the action logs as well as inline.

The definition of errorCommandFor could probably be improved to give better annotations overall, but the current definition is good enough. Here is an example of it in action.

Options

I tried a couple of things, going forward I think there are technically 3 options:

1. Enable it by default for the checks formatter (and possibly others)
2. Have a separate github formatter in hspec-core (this PR)
3. Provided it as a separate package

I'm not too eager to do (2) at this point. Basically because once we add it, we will have to maintain it indefinitely. This leaves us with (1) or (3).

Preferred Option: 1+

Initially, my preferred approach is (1) with a minor modification. Instead of emitting both, the current Failures:-list and ::error-commands, enhance the Failure:-list.

Specifically, when GITHUB_ACTIONS=true, instead of emitting

Data.String.Strip
  strip
    removes leading and trailing whitespace [✘]

Failures:

  test/Data/String/StripSpec.hs:10:7:
  1) Data.String.Strip.strip removes leading and trailing whitespace
       expected: "foo bar baz"
        but got: "foo bar"

emit

Data.String.Strip
  strip
    removes leading and trailing whitespace [✘]

Failures:

  test/Data/String/StripSpec.hs:10:7:
::error file=test/Data/String/StripSpec.hs,line=10,col=7::/Data.String.Strip/strip/removes leading and trailing whitespace/%0Aexpected: "foo bar baz"%0A but got: "foo bar"%0A

Lets call this option (1+).

Limitations of Option (1+)

Going with (1+) means that we want to colorize things within ::error (e.g. diffs). From what I tried, this kind of works:

However looking at the summary

or the annotated diff

we see uninterpreted escape sequences.

As I understand it, we need to compromise somewhere, say choose between:

a) Have redundancy in the build log (having both the current Failures:-list with colored diffs + ::error... without colored diffs) b) Only go with ::error.. and compromise on color diffs c) Compromise on the summary / annotated diff, accepting that we have uninterpreted escape sequences

As for (a), I'm not sure. If a build fails I want to see the error as quickly as possible in a representation that I can parse as quick as possible. On the one hand, having the errors in the summary/diff view could help with that, on the other hand I might want to see a colorized diff and end up scrolling through the log anyway, except for, now the log is longer, being cluttered with redundant ::errors. So it's not a clear win.

Not eager on (b), as I think colorized diffs help a lot with certain types of errors.

If we only care about functionality, then I think (c) is our best bet. But it looks broken, so :-1:.

Going forward

I think we should dogfeed option (a) for a while, to see if this is just a novelty, or if it improves productivity in a meaningful way (looking nice alone is not good enough, at least not when it comes at a cost). I think the best way to do this is, to go with (3) (create a separate package and decide later if we want to fold it into hspec proper).

Right now I'm working hard on getting 2.10.0 out. Once 2.10.0 is released, using third-party functionality with hspec-discover will require less boilerplate (https://github.com/hspec/hspec/pull/641). So I think this will come in handy with dogfeeding.

Other options

In a perfect world, GitHub Actions would allow us to colorize stuff within ::error:.. in a way that does not screw up the summary / diff annotations. I think it's worth a try to contact GitHub support about this. I'm not doing this myself right now, as my priority is on 2.10.0. @parsonsmatt @avieth if one of you guys wants to step in, that will be great!

I personally think option 2 is the best.

I'm not too eager to do (2) at this point. Basically because once we add it, we will have to maintain it indefinitely.

True, but what are the scenarios? Github does a breaking change to the format? Unlikely, but if that happens, users can just turn back to -f checks and still get a working hspec until a fix is released. Somebody can @ me and I'll be happy to do the work.

I'd like to get us back upstream. Right now we're stuck on this commit for hspec-core, which is blocking other upgrades. If we could get this merged, that'd be awesome. If you don't want the potential maintenance burden, then we can do option 3 with this. Do you have a pointer on making an external package for a formatter like this?

Do you have a pointer on making an external package for a formatter like this?

@parsonsmatt you are using hspec-discover, right?

Make sure that you read and understand https://hspec.github.io/hspec-discover.html#spec-hooks.

Start with:

module SpecHook where

import Test.Hspec
import Test.Hspec.Api.Formatters.V3

withGithubActionFormatter :: Formatter -> Formatter
withGithubActionFormatter formatter = formatter {
  formatterItemDone = \ path item -> do
    formatterItemDone formatter path item
    emitGithubActionAnnotation path item
}

emitGithubActionAnnotation :: Path -> Item -> FormatM ()
emitGithubActionAnnotation path item = write $ show (itemLocation item) -- adjust this to your liking

github :: (String, Formatter)
github = ("github", withGithubActionFormatter checks)

hook :: Spec -> Spec
hook spec = modifyConfig (useFormatter github) >> spec

Also read https://hspec.github.io/extending-hspec-formatter.html. It describes how to implement a formatter from scratch using Test.Hspec.Api.Format.V1, so it's not exactly the same thing, but some of it is still transferable.

Great, thank you so much!

We are using hspec-discover, but we have our own TestMain that does some preprocessing for us.

If we could get this merged, that'd be awesome.

I laid out my perspective in detail above. But again, briefly, before upstreaming this, this are the things I want to see:

1. Contact GitHub support and see if they have something regarding ANSI sequences in error annotations on their internal road map, or if this is something they would consider
2. External package so that we can dogfeed it / experiment with it first.

we have our own TestMain that does some preprocessing for us.

Move it into a beforeAll_ in test/SpecHook.hs if you can.

Ok, cool! I think we are going to go with the external package approach then. Feels like the easiest way to get it integrated. Thanks for the work in making the formatting extensible!!