Specify how consumers should handle parse errors

[jonathanKingston]

With a new version of TAP brings the ability to make stricter the behaviour around parsing and consuming TAP.

I think that (quick notes please feed back):

• Implementations should fail hard and fast

  • We may likely want to specify a fatal error type behaviour which consumers can choose to do
  • Failing hard should be the default behaviour
  • Implementations may however choose to ignore fatal errors but they still should be reported with the same importance as other test cases
  • Fatal errors should have set meanings codes which are human readable

• Implementations should only support the version they were designed for, when encountering a new version of TAP, the test should fail

• Reporters should also have a set of fatal errors that can be appended to their output which means something went fatally wrong with the tests but perhaps not within the tests themselves (Lets say a network error on the socket)

  • This could be something as simple as:

   TAP version 14
   1..10
   ok 1 - reason
   not ok fatal
  

  and/or interpretations might be expected to provide an out message at the end to confirm all was ok

   TAP version 14
   1..1
   ok 1 - reason
   ok complete
  

  These could be treated as less fatal than a bail out - however we could specify more around that instead.

[jonathanKingston]

Some of these comments likely clash with some of the more ambiguous comments in the: http://testanything.org/philosophy.html

This is not very helpful as I don't find any of the forwards compatible notion of TAP easy to follow as the format is so open. If this philosophy is to be followed then we need to specify further more on how libraries should behave around the edge cases caused.

As previously mentioned with other producers, this philosophy perhaps isn't ever followed anyway despite its good intentions.

[Leont]

Implementations should fail hard and fast

I suspect pragmas (#6) could be helpful here.

Implementations should only support the version they were designed for, when encountering a new version of TAP, the test should fail

Backwards compatibility is a must-have, but forwards compatibility has always been a strength. I'm not sure there's really a to disallow that, but I'm not sure of the reverse either.

Reporters should also have a set of fatal errors that can be appended to their output which means something went fatally wrong with the tests but perhaps not within the tests themselves (Lets say a network error on the socket)

I'm not sure I get your point here. Why would that be useful? Reusing ok for that seems highly confusing.

[jonathanKingston]

Backwards compatibility is a must-have, but forwards compatibility has always been a strength. I'm not sure there's really a to disallow that, but I'm not sure of the reverse either.

A pragma could be used for this, I would prefer strict but as that has some meaning already we would be safer opting for another. Personally my paranoia would prefer the ability to lock to the version I am expecting to parse in case of breaking functionality in the future or a misbehaving producer that decides to invent its own 14.5 format perhaps.

My idea around producers having fatal errors was to allow for the YAML document if there was more structured data that could be provided. Perhaps we can just have the following document after the bail out! and provide meaningful standard error codes for use in the YAML. The ok complete was similar to the report produced by TAP/Y in that it would allow consumers to check that it received all its output.

The other thought I had would be that we could specify error conditions for consumers, such that:

 TAP version 14
 pragma +always-complete
 1..1
 ok 1 - reason

could be treated as if it read:

 TAP version 14
 1..2
 ok 1 - reason
 not ok fatal 2 - consumer did not encounter a complete statement using 'always-complete' pragma 

[jonathanKingston]

Scrap that, lets provide meaningful error codes for tests only.

I think there is value to adding YAML after the bail out however, like:

...
ok 2 - reason
Bail out!
time: ...
test-running: 3
trace: ...

[Leont]

The other thought I had would be that we could specify error conditions for consumers

I think I like that idea (though in this particular case it's only useful if the plan comes at the start).

[Leont]

I think there is value to adding YAML after the bail out however, like:

Possibly it's useful to add it to the version or plan statement too. Or just a general purpose keyword for meta-information about the test run.

[jonathanKingston]

The other thought I had would be that we could specify error conditions for consumers

I think I like that idea (though in this particular case it's only useful if the plan comes at the start).

Pretty sure the pragmas will need to be always be above the tests they relate to (I shall add this question to #6).

However my thinking around this is that because we can't really control how tests are output to whatever the reporting interface is being consumed by there is the ability in the specification to explain how the consumer should treat certain failure cases and I think considering them as other tests could be the way forward (Certainly it would help easier documentation around error handling).