Runtime validation

[tonyxiao]

What's the recommended way to perform runtime validation in this library? To ensure that as APIs change or worse diverge from the spec they produce we get either warning or error?

[tonyxiao]

I know this library is intentionally trying to be lightweight and with zero runtime bundle size, so it might be understandably out of scope. However I'm looking for recommendation / discussion on which direction to look when runtime validation is relevant.

[WickyNilliams]

There are a bunch of libraries which can convert TS types to runtime validators.

There are some which do one to one mapping like ts-to-zod https://github.com/fabien0102/ts-to-zod.

Typebox is an interesting option, it is like zod but focused on json schema (on which OpenAPI is based). As well as its own validation, it has an ecosystem of tools around it, plus its own tool for code gen. It can take TS in and output zod/io-ts/valibot and a bunch of others https://github.com/sinclairzx81/typebox#typebox-codegen

With some scripting you could feed the types generated by openapi-typescript into such tools

[drwpow]

This is a great question! It’s the opinion of this library that runtime validation is a bad idea to do in general in JS. In most cases it’s a waste of resources and slows down your application unnecessarily, whether you do this in the client or on a Node.js server (and it’s important not to underestimate how slow runtime validation can be in many cases). When runtime validation is baked into your tools, you introduce potential performance bottlenecks that if your API is returning responses consistently yield no returns, and that become more difficult to rip out as time goes on. Compare this to static type inference which is “free” and can get people farther than they think.

But to answer your question “how can I know my API is returning the actual responses it says it is?” is a valid question! And there are many ways to tackle that, that don’t involve runtime validation:

1. Unit testing

Testing your actual API in a unit test is a good idea. Ideally on the API side so you don’t have as many flaky tests (but also, if your API is shooting for 100% uptime, you theoretically should be able to hit it from any remote test suite and any failure would indicate an outage). This should be the “first line of defense” against known failures.

As @WickyNilliams suggested, many zod-based tools shine here, and using a validation library within tests is highly-recommended. Test suites are resources allocated for checking the shape of your API, and your runtime app is free of this performance burden.

2. Exception capturing

Say your endpoint returns a bad shape not expected by the client. Client throws an error. If you have exception monitoring in your frontend, you’ll get notified of a breakage. This is actually good, because this counts as an outage that should be responded to. You also don’t need runtime validation here—it’s just handled by the exception capturing tool.

Also think about what’s actually used by your app. Do you care if an API was missing part of its response you were never using anyway? Probably not. You only care about the mismatches that throw exceptions. Runtime validation is probably always doing more work than needed, and throwing more errors than you need to treat as an actual incident.

3. E2E tests

Running real E2E tests against your API that run on some regular intervals is also a great way to incorporate a zod-based tool to validate API responses as well. This is basically just triggering exception capturing to happen proactively, before a user sees it. Of course, E2E tests come with their own tradeoffs that are unrelated to your question, but if you maintain an E2E suite anyway, this is a perfect fit.

---

So put another way, asking “how can I know my API is upholding its contract?” is a problem we all have to deal with, and it’s a tricky problem to solve that depend on so many factors. But if it’s bad enough you don’t trust most responses from an API, then that problem should be identified & fixed upstream, as close to the API as possible. And probably not on the client, on every request.

[tonyxiao]

Thanks for the super thoughtful response here @drwpow . I think an underlying premise here is that API is under the user's control, but it's definitely not the case for us where we spend a lot of time working 3rd party APIs of varying level of quality / assurances where we write unit test in the API codebase and e2e tests is also quite difficult to do.

I think what you say regarding exception capturing is quite relevant though. Indeed runtime validation will do a lot of unnecessary work and have spurious errors. However one concern is that sometimes exception may be raised much later in the code path where it is difficult to track down the source of the error being a misbehaving API. Having optional runtime validation that can either error or warn feels like it could still be quite valuable

[drwpow]

Yeah if the API is unreliable almost by design (user control) then I think you do have different concerns than most, and it’s up to you how best to handle that. Also, optional runtime validation seems like a good fit here—just limit it to code paths that seem to really need it.

[tonyxiao]

Gotcha. Do you see an approach that can work together nicely with openapi-fetch and openapi-typescript? I was thinking of instead of using something like zod like others try to do (I assume inference is super expensive and it's clear you've done a ton of great work here making he ts types really ergonomic and performant), to just use the underlying openapi spec to validate against request / responses in the openapi-fetch layer using something like ajv. This way it still builds upon all the great work in this library. What do you think of this approach? Is there a good openapi validation library you'd recommend? And would you like to see a PR that adds optional runtime validation / warning to openapi-fetch?

[drwpow]

Do you see an approach that can work together nicely with openapi-fetch and openapi-typescript? … Is there a good openapi validation library you'd recommend?

Not really, because openapi-typescript by its design generates runtime-free static types, and only static types. In order for runtime tools like zod to work, there has to be some executable code that creates your schema in memory, and gives the tool something to evaluate against (which isn’t possible to do from a TS interface).

While I suppose you could find some way to translate static TypeScript types to zod (e.g. tozod), it’s probably more work to do so than just building a tool that transforms your raw OpenAPI schema JSON to a zod object and going from there.

There are also tools like openapi-zod-client which seem promising, but I haven’t tried them myself.

And would you like to see a PR that adds optional runtime validation / warning to openapi-fetch?

No. For better or worse, the design of this project is decidedly within static, runtime-free typechecking. While I believe this is a good default for most people and most projects, not everyone falls into a single bucket and that’s OK! There’s never “one true way” to do things, and there are always edge cases and concerns not considered. But by placing this boundary on the project, it helps keep it well-scoped and maintainable.

So to that end, please let me know what you end up using / liking so we can recommend it to others in the docs that have the same problem!

[WickyNilliams]

By the way I used openapi-zod-client prior to deciding to migrate here. I contributed to the project a number of times (I think I have maintainer rights even).

I found that I had to disable runtime validation there because it was too sensitive, and the zod schema generation couldn't quite handle some constructs in my schema. Also I started getting TS errors maximum depth type errors, because as mentioned above, the inference is expensive. And because my API was only getting started it didn't bode well for long term prospects.

In my instance the API is developed in house, and the schema is generated as an output of the build (i.e schema last development). So I can put quite a lot of faith in the schema being correct. This is why I didn't mind "losing" the option of runtime types when migrating here.

[tonyxiao]

Not really, because openapi-typescript by its design generates runtime-free static types, and only static types. In order for runtime tools like zod to work, there has to be some executable code that creates your schema in memory, and gives the tool something to evaluate against (which isn’t possible to do from a TS interface).

@drwpow I was actually talking about the opposite of this. Which is not using zod, but using the underlying openapi spec & jsonschema itself along with a json schema validator tool such as AJV that focus exclusively on validating with no static typing to complement the work in openapi-typescript. This also avoids the problem you mentioned @WickyNilliams where the expressivity of zod is not the same as json schema and removes a potential source of error from ill-maintained json schema -> zod transformation code.

[tonyxiao]

I tried openapi zod client and migrated here too because I think the underlying mechanism of zodios is not very performant, and yes it also sometimes output zod files that don't actually compile :(

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. If there is no activity in the next 7 days, the issue will be closed.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale. Please open a new issue if you believe you are encountering a related problem.