Verify how Dredd deals with non-2xx responses in OpenAPI 3

[honzajavorek]

Dredd skips non-2xx responses for OpenAPI 2:

• https://dredd.org/en/latest/how-it-works.html#id12
• https://dredd.org/en/latest/how-to-guides.html#id13

A similar behavior should be implemented for OpenAPI 3 as well. Currently the code only deals with OpenAPI 2.

[artem-zakharchenko]

Do I get it correctly that it's about extending this conditional skipping in TransactionRunner?

https://github.com/apiaryio/dredd/blob/f7e5ee45bccd1097a65c68884cfefff250a859ca/packages/dredd/lib/TransactionRunner.js#L344-L355

Can we assert that the API description document is OpenAPI 3 by checking:

openapi: "3.0.0"

We'd need to see what toTransactions returns:

https://github.com/apiaryio/dredd/blob/f7e5ee45bccd1097a65c68884cfefff250a859ca/packages/dredd/lib/Dredd.js#L251

[honzajavorek]

I'd say the steps are:

1. Learn about OpenAPI 3 and see whether the way it generates transactions is similar enough to OpenAPI 2 so that it makes sense to approach the underlying problem the same way.
2. If yes, then write tests verifying the desired behavior.
3. Then the media type in transaction.apiDescription.mediaType should give you relevant media type for the OpenAPI 3 (a different one, not containing swagger, that's why it probably doesn't work correctly now). See the OAS3 adapter code.

The underlying problem this tries to solve is that OpenAPI 2 generates transactions in big amounts (especially through produces and consumes) and a lot of them might be for 4xx status codes. People usually don't want to test those, or in other words, they usually cannot be tested out of the box, without hooks. That's why Dredd skips non-2xx transactions by default and lets users to unskip them in hooks. That's as close to the behavior most of the people would expect out of the box as possible.

[hedleysmith]

Dredd seems to be working fairly well for OAS3 for me, except for this issue (so far). I've started a PR which gets things working for OAS3: https://github.com/apiaryio/dredd/pull/1885

[IvanGuardado]

I'm having this problem and I'm figuring out how to solve it, since I think it's important to validate all API responses, not just 2XX.

What do you think about linking requests and responses by the example id? For example:

responses:
  '200':
    description: OK
    content:
      application/json; charset=utf-8:
        schema:
          $ref: '#/components/schemas/Token'
        examples:
          example-200:
            value:
              token: "SERVER_TOKEN"
              refresh_token: "SERVER_REFRESH_TOKEN"
  '401':
    description: Unauthorized
    content:
      application/json; charset=utf-8:
        schema:
          type: object
          properties: {}
        examples:
          example-401:
            value:
              error_code: invalid_credentials
              error_msg: The provided credentials are not valid.
requestBody:
  content:
    application/json:
      schema:
        description: ''
        type: object
        properties:
          email:
            type: string
            minLength: 1
          password:
            type: string
            minLength: 1
        required:
          - email
          - password
      examples:
        example-200:
          value:
            email: [email protected]
            password: 12345678
        example-401:
          value:
            email: [email protected]
            password: 12345678
  description: ''

In the example, there are 2 request examples (example-200 and example-401) that could be matched with the response examples with the same name.