Improve Reporters Documentation and Future Vision

[bratekarate]

This may be the wrong place to ask, but I am really confused as to were this project is heading in terms of log output and reporters.

The --tape flag seems not be loved very much and to be kind of a temporary workaround? At least that's how it sounds in issues comments such as https://github.com/avajs/ava/issues/2219#issuecomment-523333284. It is not mentioned at all in the main documentation of AVA that --tape is not planned to be part of AVA in the future. Which is fine, but the issue does not really address any alternatives. E.g. what is supposed to be the workaround if the --tape flag will not be supported anymore at some point? Is the --tape flag guaranteed to stay until a machine readable output is implemented? What if a project depends on this functionality?

I am perfectly fine with the design choices, but I fail to find any concepts described how the Ava reporting is supposed to work in the future. In the issue at hand, nothing is described about JSON at all, but in https://github.com/avajs/ava/issues/2583#issuecomment-695195481, ideas about JSON output to be passed to other tools is discussed.

As a new user, this is quite confusing. I started using --tap to generate reports and after a long time researching I have still no clue if this is the right horse to bet on. And if not, what is the alternative? Writing your own parser specifically for AVAs output that generates HTML? Is there any standard that the output follows? It seems to be fruitless, as the Reporters are apparently subject to change anyway.

I think it should be much clearer where AVA is headed right now so it is easier to make a decision if it is feasible to use it in a project. I agree that reporting does not need to be part of the testing library, but it seems as though currently, the options AVA provides to be combined with other tools are kind of clumsy and deprecated (most "tape to html" tools have not been maintained for years).

[tymfear]

So the JSON reporter issue is here #2608 .

[bratekarate]

That is true, I have seen this issue. But it's still not clear to me if this is intended to be highly customizable or just one format, like the current tap reporter, also if it has any priority and what the general roadmap regarding reporters is.

[novemberborn]

Thanks for raising this @bratekarate!

The TAP reporter won't go away suddenly. Most likely its replacement will be introduced in a minor release and the TAP output will be reimplemented on top of that, so even after it's removed from AVA itself you'd be able to convert to it.

I'm not opposed to making our TAP output more complete either, though I'm not sure how much we can add that is part of the "standard", and how much would end up being some other proprietary format.

We don't have enough committed maintainer capacity for a roadmap to make sense. Things happen when somebody takes them on.

[izelnakri]

Hi @novemberborn just came across this issue, as I was going to create a similar one.

First of all, currently there is a massive open source activity happening on github since most people are at home, so excuse me if my response just adds to this friction, drains your or other maintainers mental battery :D I just want to share some thoughts.

I'd really appreciate it if eventually ava moves to TAP output by default and introduces a tap reporter. Currently I'm implementing something like this in my qunitx project. This is what I mean:

ava | npx tap-ava-reporter

It would be great because currently most tap reporters on npm currently are old, not well maintained, and lacks proper/great diffing ava already has. Thanks for your attention and your work on ava ;)

[novemberborn]

@izelnakri,

I'd really appreciate it if eventually ava moves to TAP output by default and introduces a tap reporter.

AVA has a TAP reporter. --tap.

It would be great because currently most tap reporters on npm currently are old, not well maintained, and lacks proper/great diffing ava already has.

My impression is that TAP is an older format which doesn't have well defined ways of communicating things like our diff structure. Hence my desire to have an intermediate format which represents AVA's capabilities, which can then be transformed by third-party programs into other formats with lower fidelity.

Consequently AVA's default output won't be TAP.

[izelnakri]

AVA has a TAP reporter. --tap.

Yes Im aware, and was still suggesting that we use TAP as default without flag and separate the reporter code/logic/dependencies from the main ava package :)

My impression is that TAP is an older format which doesn't have well defined ways of communicating things like our diff structure.

I'd like to know what specific data structure for diffing display we use that would be a challenge in TAP format. Here is an output that is TAP conforming:

ok 1 Some Test Module | assert true works # (1 ms)
after result
after wait
not ok 2 Some Test Module | async test finishes # (107 ms)
  ---
    name: 'Assertion #1'
    actual: null
    expected: true
    message: 'failed, expected argument to be truthy, was: false'
    stack: '    at Object.<anonymous> (file:///home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:23:12)'
    at: '/home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:23:12'
  ...
  ---
    name: 'Assertion #2'
    actual: true
    expected: 22
    message: null
    stack: '    at Object.<anonymous> (file:///home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:24:12)'
    at: '/home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:24:12'
  ...
  ---
    name: 'Assertion #3'
    actual: true
    expected: 33
    message: null
    stack: '    at Object.<anonymous> (file:///home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:25:12)'
    at: '/home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:25:12'
  ...
  ---
    name: 'Assertion #5'
    actual: null
    expected: true
    message: null
    stack: '    at file:///home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:30:14'
    at: null
  ...
ok 3 Some Test Module | another passing test  # (1 ms)
not ok 4 Some Test Module | deepEqual true works # (2 ms)
  ---
    name: 'Assertion #1'
    actual:
      firstName: Izel
      lastName: Nakri
    expected:
      firstName: Isaac
      lastName: Nakri
    message: null
    stack: '    at Object.<anonymous> (file:///home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:38:12)'
    at: '/home/izelnakri/Github/qunitx/tmp/test/failing-tests.js:38:12'

# tests 4
# pass 2
# skip 0
# fail 2
# duration 134

When we return a error yaml format, the output doesnt seem to be specified in TAP indeed, this is probably due to data format expression differences between programming languages, although they could have done a better job specifying/locking key/values at least. However this decision would just an extension on TAP even with a custom error YAML structure, the output is TAP confirming. So, Im curious what challenges we might have with diffing according to a structure like the one defined above?

Thanks for your quick response ;)

[novemberborn]

That YAML format is unspecified. It's fine for YAML compatible primitives but not for bigints, symbols or object structures. If AVA is going to emit proprietary formats, we may as well go all the way, have a high-fidelity format, and downsample to things like TAP using a separate processor.

[izelnakri]

👍 Sounds good. My remaining suggestion is to also moving the reporter code to another npm library that can process it from tap on cli execution and hold the logic importable as js inside the package. Thanks again for your responses & considerations!