IBM Validator: Better Error Handling

[thany]

Just exploring whether orval is the right tool, so I got an example openapi file (the yml version) and put it to the test. After enabling validation, which seems like a fair thing to have in any sufficiently complex application, it started spewing out an error.

I have not created a .validaterc file, for I would rather (at least for now) rely on sensible defaults, whatever they are. Besides, an openapi file is either valid or not, surely? I wonder why there is any configuration recommended, but perhaps I don't quite get it yet. But if a .validaterc is required to solve this error, please let me know what I should put in it, because IBM's documentation is extremely verbose and virtually impossible to get through for a seemingly simple example project.

Anyway, time for some code. Here's the config file:

export default {
  example: {
    input: {
      target: 'src/openapi/example.yml',
      validation: true,
    },
    output: {
      mode: 'split',
      target: 'src/openapi/example.ts',
      client: 'react-query',
    },
  },
};

And then I'm just executing orval --config ./orval.config.ts.

Here's the error I'm getting from it:

> orval --config ./orval.config.ts

🍻 Start orval v7.10.0 - A swagger client generator for typescript
🛑 example - AggregateError: Error running Nimma
    at _Scope.destroy (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:38298:17)
    at Nimma.eval (eval at query (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:40407:75), <anonymous>:535:11)
    at Nimma.query (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:40408:75)
    at execute (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:41249:13)
    at Runner.run (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:41218:11)
    at Spectral2.runWithResolved (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:48064:22)
    at async Spectral2.run (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:48072:17)
    at async ibmOpenapiValidator (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:49441:19)
    at async C:\Users\thany\AppData\Roaming\npm\node_modules\orval\dist\bin\orval.js:3059:9
    at async asyncReduce (C:\Users\thany\AppData\Roaming\npm\node_modules\orval\node_modules\@orval\core\dist\index.js:48961:18)

To be clear, without validation, it runs correctly and without any errors or warnings. And like I said, I'm just exploring what orval can do for us, so I don't know what Nimma even is. I do know that validating things will be making sense in our project.

In the documentation, it doesn't say about validation that I have to install anything in addition, so I have to assume it's built-in. And the funny thing is, searching for "validation" or "nimma" in the documentation yields 0 results, even though it's right here. But it does keep me from finding it anywhere else, if that.

I also couldn't find any existing issues, closed or open, about Nimma. Searching for "validation" yields only seemingly unrelated results.

[melloware]

It's trying to validate your open api schema and validation. Try turning validation off in your Orval config.

Oh I see you said it works. The issue is the IBM validator is not perfect nor is it up to date with the latest specs so when in doubt turn it off.

[melloware]

Closed as this is an IBM validator related issue so marked as third party.

[thany]

Oh I see you said it works.

Yes, it does without the validator.

Look, why are you including a validator if it doesn't work? It's a simple matter of trying it out - if it works, great. If it doesn't work - revert to whatever version that did work.

You're now brushing it it off to being a 3rd party problem, but we are still left with a non-functioning validator, and so are you. If you the version you included with Orval doesn't work, isn't the solution to roll it back to a version that did?

But if you feel you can't trust IBM do do a good job at it, maybe don't include it?

[melloware]

the validator works fine for me. But we have seen depending on the environment or OpenAPI spec we have gotten reports like yours. Maybe it should be false by default.

[thany]

I'm sure the validator can be made to work with some specially crafted spec file, but that's not the point. The point is, can we make it make sense? The error is extremely obscure.

If the validator is unreliable, turning it off by default is sensible. Removing support for it also seems sensible to me. But spewing out an error like that, that only causes more confusion, and doesn't quite feel like the right thing to do. So maybe it should display a warning "Validator crashed, continuing without validation..." or something.

[melloware]

Agreed are you able to provide your OpenAPI file that causes it to crash so we can reproduce and try and have a better error message?

[melloware]

Actually the code that is failing is right here: https://github.com/orval-labs/orval/blob/1626b34f31927d5dd6d62f2ab70668280df81920/packages/core/src/utils/validator.ts#L24

Its running the spectral in async mode. Do you want to submit PR somehow adding error handling around this and giving a better error message?

[melloware]

I did find this issue: https://github.com/stoplightio/spectral/issues/2053

it seems Nimma is a library that IBM Validator uses.