Validate examples against the schemas

[m-mohr]

Detailed description

It would be very helpful if speccy could validate the examples against the schemas so that we can ensure they are valid. We sometimes change the schemas for a good reason, but fail to fully adapt the examples. This would lead to an overall better experience when consuming the specification.

Context

I think this is a valuable extension for many users as it helps to make complete and valid examples. Would be very helpful in CI to make sure PRs and changes in general are adopted throughout the whole API specification.

[pderaaij]

I am planning to add a load of integration test, testing rules individually. I just await the current PRs to be merged

[m-mohr]

Thanks for the response, but wait, I was speaking about the examples in the openAPI files that are validated by speccy. From your response it seems to me you understood it differently?

[pderaaij]

No, sorry. I also like to add that. But, there are already some simple tests (see https://github.com/wework/speccy/blob/master/test/integration/lint.test.js) but that still doesn't catch all the things.

Creating tests on rule level makes it more granular.

[cbautista1002]

I would also find this feature to be useful for the same reason as posted by @m-mohr.

@pderaaij Here's an example:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  description: Swagger Petstore
  license:
    name: MIT
  contact:
    name: Test
servers:
  - url: http://petstore.swagger.io/v1
tags:
  - name: pets
paths:
  /pets/{petId}:
    get:
      summary: Info for a specific pet
      operationId: showPetById
      description: Get a pet
      tags:
        - pets
      parameters:
        - name: petId
          in: path
          required: true
          description: The id of the pet to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Expected response to a valid request
          content:
            application/json:
              schema:
                type: object
                properties:
                  id:
                    type: integer
                    example: "1"

The example above is a string, but the type of id should be integer.

It would be great if speccy could detect this invalid example.

[pderaaij]

Ah, I get what the purpose is.

In order to support this it looks like we need a new type of rule in oas-linter. As far as I can see right now is there no way to apply a rule that looks at the value of a different property of the scheme.

I will have a look into this in a later moment or perhaps @MikeRalphson has an idea on this?

[philsturgeon]

Spectral does this by default if you folks want to use that for inspiration.

https://github.com/stoplightio/spectral

It’s the “valid-example” rule, which is in both oas2 and oas3.