Option to retry from a previous step on failure of step

[a1shadows]

@authorjapps

In situations where we need to retry a step that is dependent on a previous step, for eg. an api call that uses the previous step to generate auth, when the step is retried on failure, it should retry from the previous step.

I can look into adding this as a feature

[authorjapps]

@a1shadows , makes sense. You can look at this and add a test scenario 1st to check if this feature fails and push it, let the CI build fail. If fails, then please raise a PR to fix it. Happy to merge this when you are ready.

Appreciate your great suggestions and support.

[a1shadows]

@authorjapps Do you have any suggestions on approach?

The approaches I can think of are:

1. Define a retryFrom parameter in the Step and go to the retryFrom step on failure. We will have to maintain retry count for the step we retried even though execution will have moved to a different step.
2. We can have a special type of step called MultiStep which will contain Steps within it and for failure anywhere in the multi step, the whole Multi Step will be retried. Local retries in the Steps within the MultiStep should also be feasible. A MultiStep would be very similar to a Scenario, so maybe we can also think of it like enabling retry for a whole Scenario and allow composition of multiple Scenarios together

Can you think of any other approaches?

[authorjapps]

@authorjapps

In situations where we need to retry a step that is dependent on a previous step, for eg. an api call that uses the previous step to generate auth, when the step is retried on failure, it should retry from the previous step.

I can look into adding this as a feature

Apologies, I couldn't get the problem statement correctly.

Can you please give an example to explain it? A scenario example with 2 steps should be easy ok I think.

Thanks

[a1shadows]

I'm talking about a scenario like this:

{
    "scenarioName": "Create employee by authenticating via Oauth2 token",
    "steps": [
        {
            "name": "auth",
            "url": "com.zerocode.authClass",
            "method": "authMethod",
            "request": "",
            "assertions": {
                "token": "$NOT.NULL"
            }
        },
        {
            "name": "create_emp",
            "url": "/api/endpoint",
            "method": "POST",
            "request": {
                "headers": {
                    "token": "${$.suth.response.token}"
                },
                "body": {
                    "id": 1000,
                    "name": "Larry Pg"
                }
            },
            "retry": {
                "max": 3,
                "delay": 2000
              },
            "assertions": {
                "status": 201
            }
        }
    ]
}

Now, let's say the auth has a TTL of 2 seconds and the API call step fails and goes into retry: Since the auth has already expired, the second step is now doomed to fail no matter what.

So, if we want to retry the second step, ideally we should be able to retry from the 1st step so that a fresh auth is generated for each time the api call step is executed.

[authorjapps]

Now, let's say the auth has a TTL of 2 seconds and the API call step fails and goes into retry: Since the auth has already expired, the second step is now doomed to fail no matter what.

So, if we want to retry the second step, ideally we should be able to retry from the 1st step so that a fresh auth is generated for each time the api call step is executed.

Understood. Thanks for explaining it.

Yes, in theory, this is possible, but in reality I think it would be very very rare. This is because in test environments, the auth-token mostly(normally) kept for bit longer time so that the token is reused.

Are you really facing this problem in your current project i.e. in test envs ?

[a1shadows]

@authorjapps Hey, yes, we faced this problem at my workplace. So, the use case is something like this:

There are 2 APIs:

1. A create request POST API: this initiates the creation process for the request on the backend. This process can take anywhere between 1 second to 20 seconds depending on the type of request. However, the API returns success immediately with a request_id for future use
2. A status check API: This can be used to check the current status of a previously submitted request using status id.

Both APIs use a short lived dynamically generated auth strategy. Now, what I need is the capability to do a polling loop on the second API call in my test scenario with some poll interval and a timeout (similar to how the awaitability libs for java testing work).

For this, I need the capability to retry both the auth generation step and the API call step with every retry.

[authorjapps]

@authorjapps Hey, yes, we faced this problem at my workplace. So, the use case is something like this:

There are 2 APIs:

1. A create request POST API: this initiates the creation process for the request on the backend. This process can take anywhere between 1 second to 20 seconds depending on the type of request. However, the API returns success immediately with a request_id for future use
2. A status check API: This can be used to check the current status of a previously submitted request using status id.

Both APIs use a short lived dynamically generated auth strategy. Now, what I need is the capability to do a polling loop on the second API call in my test scenario with some poll interval and a timeout (similar to how the awaitability libs for java testing work).

For this, I need the capability to retry both the auth generation step and the API call step with every retry.

Got you @a1shadows.

Note: "method": "POST" ==> might create another employee on retry, which may be unwanted.

Anyway, have a thought once again and then, if you really need, my suggestion would be to go via following way to use withStep (instead of fromStep ).

Example:

    "max": 3,
    "delay": 2000,
    "withStep" : "auth"  <---- Step name which should be executed 1st before this Retry block
}

I think for your requirements, an optional key withStep sounds better 🟢 , because you want to execute that n'th step, but execute the "auth_step" 1st (instead of executing all the steps from "auth_step" onwards 🔴 ) and use the output of that "auth_step" in this n'th step.

---

instead of executing all the steps from "auth_step" onwards 🔴

Because, you could have in some situations more steps in between the n'th step and "auth_step" and executing all other steps are not necessary.

step1 : auth_step
step2 : create_emp
step3 : get_emp

// or

step1 : auth_step
step2 : create_emp
step3 : create_identity
step4 : get_emp  <------ By this time "auth token" is expired

Hence, executing "auth_step" prior to "get_emp" would make sense. 
Also you don't have to execute "create_emp" and "create_identity" steps in this scenario.

[a1shadows]

@authorjapps Can we make withStep accept a list? That way just in case we need to execute multiple steps to get the auth, we can execute all of them. Something like "withSteps" : ["auth_step1", "auth_step2"] ?

The steps will be executed in the order specified

[authorjapps]

yep, I think that would be much better and this will accommodate various different usecases too. 👍

{
    "max": 3,
    "delay": 2000,
    "withSteps" : ["auth_step1", "auth_step2"]
}

The steps will be executed in the order specified

Yep, order is important

[a1shadows]

@authorjapps Hey, I'm trying to figure out how to implement this feature and there are quite a few things to consider and work out.

What is the purpose of StepExecutionState and ScenarioExecutionState? I was planning to use these to get the previous step execuation status and replace the new execution state in ScenarioExecutionState for the steps in withSteps that were re-run. However, I am unable to understand the exact semantics and purpose for these classes. I need to be able to replace exectution info of a step in ScenarioExecutionState as well as fetch a specific step's StepExecutionState from it.

I am currently planning on making the changes in the executeRetry method in the ZeroCodeMultiStepsScenarioRunnerImpl class in the retry for loop.

There also need to be some validations to make sure that the user does not submit a future step in the withSteps parameter and that it is a step that was successful prior to this step.

Do you have any suggestions?

[a1shadows]

@authorjapps I came up with a Draft of changes(PR linked to this issue) to StepExecutionState and ScenarioExecutionState that would be super useful while implementing this feature. Some minor changes will be required in a few test which I can take up.

The idea is to make it easier to mutate ScenaioExecutionState in complex retry scenarios such as cases where previously successful steps need to be rerun.

Another alternative can be to consider the retried steps as new steps with "-retry-{retriedStepName}" appended to the step name. Can you take a look?

[a1shadows]

@authorjapps I've implemented a working approach for this in the linked PR. Can you take a look?

[authorjapps]

@authorjapps I've implemented a working approach for this in the linked PR. Can you take a look?

Thanks @a1shadows, was slightly busy with my day job,

As the CI is Green 🟢 , hoping all is well, but will have a look soon anyways. 👍