When a catastrophic error happens the output of tap-arc is success

[telmotrooper]

trafficstars

Describe the issue When a catastrophic error happens the output of tap-arc is success. It should be an error, just as if a test failed.

Steps to reproduce Steps to reproduce the behavior:

1. Add anything to a test that would throw a catastrophic error such as an invalid import: const { hello } = require('./this-is-a-fake-path')
2. Run tap-arc and check how even though you get errors, the output still returns success, which would make a pipeline step assume nothing went wrong with the tests.

Expected behavior When a catastrophic error happens, the output of tap-arc should indicate failure (return 1).

Screenshots (Green arrow in Oh My Zsh indicates successful execution of last command, while red indicates failure.)

Desktop Please complete the following information (if appropriate):

• OS: Manjaro Linux
• Version: 0.3.3

[tbeseda]

Good catch, thanks for reporting!

I did some research and interestingly there isn't a great way to get the exit code (or any sort of state) from the previous command in a pipeline. Seems like the safe thing to do is assume that if there are zero test results, something is busted and we should exit 1. Looks like this is how other reporters handle this case.

I'll release 0.3.4 shortly with the fix.

[Powersource]

Been running into a few situations with the latest tap-arc where tape exits with 1 but tap-arc exits with 0.

% npm t

> @ssb-graphql/[email protected] test
> npm run test:js && npm run lint


> @ssb-graphql/[email protected] test:js
> npm run test:raw | tap-arc


> @ssb-graphql/[email protected] test:raw
> tape -r esm test/**/*.test.js


saveArtefact (create + get)
    ✓ save artefact type document doesnt throw errors
/home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/errors/errors.cjs:31
        var _this = _super.call(this, errorMessage) || this;
                           ^

ApolloError: Abstract type "Artefact" must resolve to an Object type at runtime for field "Query.artefact". Either the "Artefact" type should provide a "resolveType" function or each possible type should provide an "isTypeOf" function.
    at new ApolloError (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/errors/errors.cjs:31:28)
    at /home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/core/core.cjs:1843:47
    at both (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/utilities/utilities.cjs:1072:53)
    at /home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/utilities/utilities.cjs:1065:72
    at new Promise (<anonymous>)
    at Object.then (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/utilities/utilities.cjs:1065:24)
    at Object.next (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/@apollo/client/utilities/utilities.cjs:1073:49)
    at notifySubscription (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/zen-observable/lib/Observable.js:135:18)
    at onNotify (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/zen-observable/lib/Observable.js:179:3)
    at SubscriptionObserver.next (/home/me/prj/ssb/ssb-graphql-artefact/node_modules/zen-observable/lib/Observable.js:235:7) {
  graphQLErrors: [
    {
      message: 'Abstract type "Artefact" must resolve to an Object type at runtime for field "Query.artefact". Either the "Artefact" type should provide a "resolveType" function or each possible type should provide an "isTypeOf" function.',
      locations: [ { line: 2, column: 3 } ],
      path: [ 'artefact' ]
    }
  ],
  protocolErrors: [],
  clientErrors: [],
  networkError: null,
  extraInfo: undefined
}

Node.js v18.16.0

total:     2
passing:   2
3.147 s


> @ssb-graphql/[email protected] lint
> standard --fix

me@me-lap[ssb/ssb-graphql-artefact]% echo $?
0

[Powersource]

I guess the difference is that in my case a few tests actually pass at first and so the 0-test check doesn't get triggered.

[tbeseda]

Apologies, I've been neglecting my GH notifications. I'll re-open this and have a look!