Testing validations with a bad example

[xocasdashdash]

Hi there!

Please let me know if this is the wrong way of asking this quest as it's more of a support question than an issue with the project (been using it for 30 min and loving it already)

I'm trying to check how the validations work, so i have an api with an already valid example. I make a request to this endpoint and it works great (all using the mock mode). Now i go and drop fields from the example that are marked as required in the api spec, at this point i'm expecting that the next request will show up as an error.

The reality is that is that it comes back ok. I've also tested by messing with the types (putting a string in a field that's defined as a bool). This also works ok.

I've been able to trigger the validations by making a bad request with a non valid body and i can see that it works as expected.

Is this the correct behaviour? i was hoping that if a server returns a "bad" response it would show up as a violation, is this because i'm running in mock mode?

[daveshanley]

Hi,

I am a little confused, when you drop fields from the example, where are you expecting it to fail? Examples are only used to render a response in mock mode, it's not used in any other way for validation, the schema is still what is used to validate requests and responses.

I need a little more information as to how you're using the tool (it's quite flexible and folks use it in lots of different ways)

[xocasdashdash]

yeah, so i'm trying to use as part of a local developer flow. The flow i'm thinking of is the following:

• Dev A writes a new API endpoint spec and adds some examples. How can he validate that the examples follow the spec?
• Assume that Dev A makes a mistake and doesn't check his examples correctly, how can Dev B use wiretap as a mock server to test the API if the specs are not validated it?

[daveshanley]

• Dev A writes a new API endpoint spec and adds some examples. How can he validate that the examples follow the spec?

I see, so this part of the puzzle is handled by another tool that does OpenAPI spec analysis. It's called vacuum and read more about it at https://quobix.com/vacuum

If you're feeling brave you can use our newest (early alpha) tool called the OpenAPI doctor, that will combine wiretap and vacuum into a suite (eventually) https://doctor.pb33f.io

I don't plan on building this functionality into wiretap, because the doctor will provide both wiretap capability and OpenAPI analysis and change detection.

[xocasdashdash]

ahh i understand, but do you think there's some difference in behaviour when you do a get request that returns an example, even though it's invalid. And when you do something like a post request that you can see a violation on the UI

[daveshanley]

ahh i understand, but do you think there's some difference in behaviour when you do a get request that returns an example, even though it's invalid. And when you do something like a post request that you can see a violation on the UI

I am struggling to understand what you mean, would you be able to explain it a little differently to me? I apologize, I work across multiple OpenAPI projects and my brain sees OpenAPI like a compiler does, I struggle a little sometimes with concepts like this.