[FEATURE] Create multiple examples for multiple tests for same error code.

[lpender]

trafficstars

Great gem, thank you for sharing!

Here's a bug and a suggested feature to fix it.

The below is pseudo-code -- apologies if it doesn't work.

Steps to reproduce:

In your rspec tests, write something like:

context "when params have an issue, returns bad request" do
  it "notifies that params are missing" do
    post "/mypath"

    expect(response.code).to eq("400")
    expect(response.body).to eq("missing required parameter: param_a")
  end

  it "notifies that params do not match schema " do
    post "/mypath", with: {param_a: 123}

    expect(response.code).to eq("400")
    expect(response.body).to eq("param_a must be a string")
  end
end

Actual result:

It seems that only the response body received for the first or last test that is run is placed into examples. Because the test order is randomized, the output is therefore not deterministic. In other words, sometimes you'll see:

...
   example:
     "param_a must be a string"

and sometimes you'll see

...
  example: 
    "missing required parameter: param_a"

Expected result:

The output should be deterministic (ordered the same way every time, i.e. alphabetically) and should include all of the examples from tests.

i.e.

...
  description: when params have an issue, returns bad request
  examples:
    notifies_that_params_are_missing: -
      missing required parameter, param_a
    notifies_that_params_do_not_match_schema: -
      param_a must be a string

If you're open to it, I can work on a PR!

Thanks again!

[exoego]

Interesting. I did not know examples that can contain multiple examples https://swagger.io/docs/specification/adding-examples/

I am happy to accept pull request.

I prefer it defaults to single example and add a feature dlag for new behavior (examples)

[lpender]

Ok great. I'm in the middle of a project with a deadline and I haven't looked at the code yet.

So I'm not sure when I'll be able to get started but I will notify here if and when I do.

Please let me know if you have any recommendations on how you would architect / implement this.

Thanks again!

[lpender]

Hey--I'm realizing that this may be a killer bug for me... The issue effects not only outputs but inputs.

I need to test that the API gives a 200 for good input and 400 for bad input--a very common use case.

Unfortunately, the requestBody example will randomly generate the OAI schema using the bad input from time to time.

Let me know if there are any workarounds or if you think this might be a quick fix.

Thank you

[exoego]

Let me know if there are any workarounds or if you think this might be a quick fix.

Test cases can be excluded from OpenAPI generation with openapi: false argument. So you can include only valid inputs.

RSpec.describe '/resources', type: :request do
  it 'returns a resource', openapi: false do
    # ...
  end
end

[lpender]

The thing is, I'd like to document 400 responses as well.

I think this will tide me over for now actually:

# Disable generating `example`
RSpec::OpenAPI.enable_example = false

Thanks !