[BUG] Explicit mode and GET request without examples

[serieznyi]

Checklist

• [x] I checked the FAQ section of the documentation
• [x] I looked for similar issues in the issue tracker

Describe the bug Schemathesis in explicit mode doesn't send any request to URL what doesn't have examples. Also test of this request marked as successful, but I see you already know about it (#1323) Maybe Schemathesis needs more documentation about phases or I don't find it.

To Reproduce Steps to reproduce the behavior:

1. Use specification what I provide
2. Run tool in CLI mode using next command schemathesis run --hypothesis-phases explicit http://example.local/open-api.yaml
3. Watch what GET /api/status marked as successful
4. Watch web server logs and make sure that request didn`t send.

If possible, please post a minimal version of your API schema that cause this behavior:

openapi: "3.0.3"
info:
  title: Title
  version: 1.0.0
paths:
  /api/status:
    get:
      responses:
        '200':
          description: Ресурс успешно получен
          content:
            application/json:
              schema:
                type: integer

Expected behavior I expect that schemathesis execute test without examples because they don`t need for request.

Environment (please complete the following information):

• OS: Manjaro Linux
• Python version: 3.9.10
• Schemathesis version: 3.12.3
• Spec version: Open API "3.0.3"

Additional context

Provided command run in docker image

FROM python:3.9.10-slim

RUN pip install --upgrade pip

RUN pip install \
                schemathesis==3.12.3 \
                mysql-connector-python

[Stranger6667]

Hi @serieznyi

Thank you for opening this and providing a detailed report! Indeed, we need better docs for phases. Schemathesis follows the Hypothesis behavior where the explicit phase executes only examples provided via the example decorator. For example, the following snippet would indicate 1 passed test, though the test's body always raises an exception:

from hypothesis import given, strategies as st, settings, Phase

@given(st.text())
@settings(phases=[Phase.explicit])
def test(x):
    1 / 0

I definitely agree that it might be a confusing UX, that can give a false sense of safety for the end-user (it indicates that the test passes, but it is actually is not executed).

I expect that schemathesis execute test without examples because they don`t need for request.

It sounds like there are multiple use cases:

• Execute only explicit examples defined in the schema
• Execute explicit examples & generate a single request when there are no examples defined

The first one is supported now, but the indication is misleading. I think it could be solved via #1323. I saw a couple of times, that Schemathesis is used exactly for the second scenario (there were a few questions on gitter as well).

So, maybe, it can be solved a bit differently - something like REST-ler does. They have multiple commands - test, fuzz-lean, fuzz which execute tests in different ways. In the case of Schemathesis, it could be something like schemathesis run-minimal or something like that which will execute tests with explicit examples if they are defined, or with a single generated request otherwise.

However, a similar effect could be achieved with --hypothesis-max-examples=1, but in this case endpoints with explicit examples will have this fully generated request unconditionally. Not sure whether it is a concern in your particular use case.

At the moment, I am not sure what would be the right approach here, surely some design work should be done before proceeding. Though, I am almost sure that #1323 & clarifying docs will improve the status quo.

Let me know what do you think :)

[serieznyi]

I want to apologize in advance for the potentially stupid argumentation because I don't work with Python and don't knew how tight relations beetwen Schemathesis and Hypothesis

I agree that #1323 must be first step for decisions this problem, because UI literally doesn't notify me about not executed test case.

What I don`t understand:

In generate phase Schemathesis generate next Case for APIOperation without any parameters/body/query in schema

Case(headers={'Authorization': 'Bearer '})

How you see Case doesn't have any generated data for any parameters/body/query because schema doesn`t have them, and Case will execute.

Maybe Schemathesis must do same in explicit phrase for same APIOperation

How I see the solution this problem:

1. In explicit phase for APIOperation without examples AND without any parameters in schema - generate empty Case and execute him.

2. Add support for define phase on APIOperation level using custom annotation x-schemathesis-phase: generate.

3. Add additional phrase what work as explicit, but also not ignore APIOperation without examples. It's almost the same as 1, but not change original behavior for explicit phrase

In this moment I use fake query parameter for my path to run case.

openapi: "3.0.3"
info:
  title: Title
  version: 1.0.0
paths:
  /api/status:
    get:
      parameters:
        - name: fake
          in: query
          description: Fake param for schemathesis
          schema:
            type: string
          required: false
          examples:
            fake:
              value: fake
      responses:
        '200':
          description: Ресурс успешно получен
          content:
            application/json:
              schema:
                type: integer
                

[Stranger6667]

Solved in #1728 by adding the --contrib-openapi-fill-missing-examples CLI option