[BUG] --hypothesis-phases=explicit, no checks performed

[hoog1511]

Describe the bug I had a schema which I was using for Schemathesis testing which had a dummy example value for a apikey header (i.e. 5140286e-65c6-435d-991c-52635c05ddb7).

When I performed tests using the CLI with the --hypothesis-phases=explicit configuration, I provided a different valid value for that header (i.e. b12b93c6-b0b6-40d2-bfef-2bd95a38666e). Once I started up the test the Schemathesis tool performed no checks, probably due to it not knowing which value it needed to use for testing due to the configuration.

To Reproduce Steps to reproduce the behavior: add a example value for a header and during testing provide a different value and use the --hypothesis-phases=explicit configuration.

Expected behavior Provide a specific error message to help find and solve the issue (either remove the example or provide no value through the CLI command). OR Let the provided value from the cli command overwrite the example during this specific testcase.

Environment (please complete the following information):

• OS: Windows
• Schemathesis version: latest

[Stranger6667]

Hi @hoog1511

Thanks for reporting! I am leaning towards the first option with an error message. The latter one feels like it will hide the fact that one of the examples is not tested at all.

[hoog1511]

Hi @Stranger6667

I recently ran a schemathesis dry run to check data generation of my OAS and noticed a peculiar bug. Having an example provided for a apikey header which is also designated as a security schema; causes example values to be ignored for a path parameter of mine. The patterns are still used but I cant see the example values anywhere in the generated data.

parameters:
  Id:
    name: Id
    in: path
    required: true
    schema:
      type: string
      pattern: '^[0-9]{15}'
    examples:
      valid:
        value: "000000000012345" # <-- being skipped over
        summary: ...

  ApiKey:
    name: ApiKey
    in: header
    required: true
    schema:
      type: string
      format: uuid
      example: "00000c00-ad00-00ec-a0ad-00ddcb0f0021" # <-- cause of issue

####################################################
securitySchemes:

  ApiKey:
    type: apiKey
    description: ....
    name: ApiKey
    in: header

I required the path parameter example to test the happyflow of a GET request which asks for an existing id in a connected database.

To fix this I had to omit the apikey example from the OAS. I attached this comment to my earlier bug report due to both of them having issues with apikey header example values.

to reproduce this error

have an OAS with both an apikey header and a path parameter. Provide an example uuid for the apiKey and perform a dry-run using the CLI tool.

From my experience the first couple results should normally be containing an example value for the path parameter (if the apiKey example is omited),

[Stranger6667]

I tried to reproduce it, but it seems to be fixed by 1bd3f99b. That fix prevents overwriting regular parameters with security definitions, and therefore after that change, the original parameter examples are not overwritten as well. The latter case with path parameters looks to be the case of #1729.

Going to close it and check examples generation for path parameters separately in #1729