[FEATURE] automatic CRUD operation on one endpoint

[dil-gfischhof]

This feature request somehow relates to #548 and #604

Is your feature request related to a problem? Please describe. we have to create several links to do operations on one endpoint

Describe the solution you'd like If the endpoint has appropriate operations, then schemathesis should test them in the following order, and with gathered data

find out some random thing get that thing -> result 404 post the thing -> result 200, 201 (accordingly - from doc)) get the thing using "ID" from post as ID, if the schema says that ID -> the random thing is got, 200 put / patch the ID -> success get the thing by ID -> got the put / patched object delete the thing by ID -> success get the object -> 404 again

this would be a very good thing, to automate API testing

Describe alternatives you've considered like in #548 the flow could be written in graph somehow adding the name of the linkage field if not ID

BR, George

[Stranger6667]

Hi @dil-gfischhof

It definitely makes sense and feels very much like links inference with some heuristics. At the moment, there are quite a lot of things I am not certain about:

• Configuration API. How the end-user would like to configure such tests?
• How generic and strict the default implementation should be?

I also don't think that the current state machine-based implementation is the right one for such things because it randomizes the tested rules and uses Swarm testing, but this one looks like something that users want to run on every endpoint every time.

At the moment it might be done with a custom interactive Hypothesis test that uses Schemathesis strategies:

from hypothesis import given, strategies as st

import schemathesis

schema = schemathesis.from_uri(
    "https://example.schemathesis.io/openapi.json",
    validate_schema=False
)

@given(data=st.data())
def test_crud(data):
    create_user = data.draw(schema["/users/"]["POST"].as_strategy())
    response = create_user.call_and_validate()
    if response.status_code == 201:
        get_user = data.draw(schema["/users/{user_id}"]["GET"].as_strategy())
        get_user.path_parameters["user_id"] = response.json()["id"]
        response = get_user.call_and_validate()
    # Take other endpoints & do more calls and assertions

For me, the main question is how to reliably detect such connections for any API definition without being too strict on what will work or not.

This direction is something I will experiment on and I'd be happy to discuss it further. It will be extremely helpful for me if you could share your endpoint example you'd like to run such tests on.

[dil-gfischhof]

Hi @Stranger6667,

yeah, sure ;) I will collect them, anonymize and share. Unfortunately our API is not a standard REST API, but as I heard from a colleague it is called functional API... So practically, these are two different things... (Also we have some additional restrictions on that particular API, for example some value must be unique, so those parts require customized, home brew solution anyway)

I think for first it would be good to be able to test standard things ;-) which is REST API.

BR, George

[dil-gfischhof]

Hi @Stranger6667,

nowadays I have no time, unfortunately... Our API looks like this:

get /v1/object/ post /v1/object/add delete /v1/object/delete

but others could have some different their own stuff, so with this we are not going forward, so standard RESTful API should be in focus at least for first (maybe we will migrate to that too)

BR, George

[fscarlato]

Hello! The suggested code gives me the flaky exception, what is wrong with it? I just copy pasted and changed the operations path.

       raise Flaky(
            "Inconsistent data generation! Data generation behaved differently "
            "between different runs. Is your data generation depending on external "
            "state?"
        )