How to Forcefully fail a step without raising an Exception ?

[eerkunt]

Question

Hello,

Gone through the issues but couldn't find one, also spend few hours on the code hacking and mocking some parts to do this, but still couldn't do it. Thus, let me ask the owner about how to do it :)

I would like to have an option where I would like to mark a step as Failed (but continue to run) without raising an Exception.

I tried ;

step.state = Step.State.FAILED
step.failure = MyOwnExceptionThatFailsConditionaly(message)

but it looks like I can't make it due to stepmodel.py structure in 0.13.1. https://github.com/radish-bdd/radish/blob/14b84219499be82b448bbd5262bcdf8a237617ad/radish/stepmodel.py#L131. Not sure, if I am pointing the right direction though.

I was going to create a PR, but found out master branch is quite different from what I have on my local, which is 0.13.1. Played with master branch a bit, but the change is quite big that I also need to change how I invoke radish within my code

https://github.com/eerkunt/terraform-compliance/blob/9d8ecb55e3fd6fcd15c463cb78f1c2f206918011/terraform_compliance/main.py#L71

Any plan supporting failing a step within a step without giving an Exception ?

Hope I could explain myself.

Thanks for the support 🎉

[eerkunt]

Just to give more information, I am currently handling this on my local by changing

https://github.com/radish-bdd/radish/blob/14b84219499be82b448bbd5262bcdf8a237617ad/radish/stepmodel.py#L151-L156

to

       else:
            if self.state is Step.State.SKIPPED:
                self.skip()
            elif self.failure is not None:
                self.state = Step.State.FAILED
            elif self.state is not Step.State.PENDING:
                self.state = Step.State.PASSED
        return self.state

[fliiiix]

did you checkout --wip ? https://radish.readthedocs.io/en/stable/commandline.html#run-work-in-progress

[eerkunt]

Hi @fliiiix,

Yes, I am using --wip in many places, but it is not fulfilling my requirement. I don't want exit code to be changed (or let's say reversed with --wip). What I need is to fail a step within the same step without giving any exceptions, and continue on execution and fail more if the test in the step is failing again - without any exception.

Let me link the issue I am dealing right now for keeping things interconnected. https://github.com/eerkunt/terraform-compliance/issues/170

Thanks 🎉

[eerkunt]

Hello,

Any suggestions/hope about this ? Since I need to release 1.1.0 as soon as possible, I think I will fork 13.1 and create a 13.1_tc version specifically for terraform-compliance. I will have a look on further stages when #391 is merged with master.

[eerkunt]

Thinking about that, it is a bad idea. Let's not create duplicate packages in PyPI for just few lines.

Will wait for your answer.

[eerkunt]

Hello guys,

It has been a while, don't want to push you really, but we are still waiting an update from you to trigger a new release.

[timofurrer]

Sorry, I couldn't give some attention earlier ;)

I would like to have an option where I would like to mark a step as Failed (but continue to run) without raising an Exception.

I'm kinda confused about the use-case here. So I'll just provide you some options I think might help you:

1. Just use regular Scenarios (without the --early-exit flag) - a Step failure will just stop the Scenario from running further, but not the Feature.
2. Make use of step.pending() or step.skip() instead of raising an exception and with that letting the Step fail.
3. Somehow write a result file / logs in the case you are currently raising an exception and evaluate that in the end of a run. You could also attach arbitrary data to a Step with step.embed() - with that this data ends up in the cucumber report - you could again, evaluate that.

Played with master branch a bit, but the change is quite big that I also need to change how I invoke radish within my code

That's right - the current master got some major refactoring. If you could give me more examples and a valid use-case for radish why it should have some kind of mode or Step state which let's a Step fail without stopping the run - we might be able to bring it into radish v1.0.0 which will be created from master later this year.

[eerkunt]

Hello @timofurrer,

Thanks for the attention :)

Let me try to explain the situation. As you may know we use radish-bdd in terraform-compliance. Normally when you create a Scenario, it executes it and fails the scenario on the first failure occurs via raising an Exception. This is stopping any further step execution on the same scenario, but it doesn't stop any other Scenario running.

The problem here is, as terraform-compliance is a compliance-as-code implementation against terraform, there are some use-cases where a step in a Scenario might fail on multiple times. Like show me all the problems in my terraform code, instead of showing one and exit on the Scenario. I resisted these feature requests as terraform-compliance never intended to be a linting tool, but it looks like I can't resist anymore :)

To explain this better, let me give an example ;

    Scenario Outline: Well-known insecure protocol exposure on Public Network for ingress traffic
        Given I have AWS Security Group defined
        When it contains ingress
        Then it must not have <proto> protocol and port <portNumber> for 0.0.0.0/0

    Examples:
        | ProtocolName  | proto | portNumber |
        | HTTP          | tcp   | 80         |
        | Telnet        | tcp   | 23         |
        | SSH           | tcp   | 22         |
          Failure: tcp/22 port is defined within 0.0.0.0/0 network in module.mysg.aws_security_group.alb.

So, based on this result, all looks ok, right ? There is a security group somewhere in the terraform code, that hits the checks implemented in the step and it fails.

The summary of the result is ;

1 features (0 passed, 1 failed)
9 scenarios (8 passed, 1 failed)
27 steps (26 passed, 1 failed)

So we have 1 failed scenario, as expected. We have an exception class called Failure and we just raise this exception within the steps and it works without any problem.

... but what if there a multiple security-groups failing on the same exact step ?

Coming with the new terraform-compliance release, we implemented to have a feature where we now report multiple failures on a step (not a Scenario!).

E.g. ;

    Scenario Outline: Well-known insecure protocol exposure on Public Network for ingress traffic
        Given I have AWS Security Group defined
        When it contains ingress
        Then it must not have <proto> protocol and port <portNumber> for 0.0.0.0/0

    Examples:
        | ProtocolName  | proto | portNumber |
        | HTTP          | tcp   | 80         |
        | Telnet        | tcp   | 23         |
        | SSH           | tcp   | 22         |
          Failure: tcp/22 port is defined within 0.0.0.0/0 network in module.mysg.aws_security_group.alb.
          Failure: tcp/22 port is defined within 0.0.0.0/0 network in aws_security_group.failure.
          Failure: tcp/22 port is defined within 0.0.0.0/0 network in aws_security_group.another_failure.

1 features (0 passed, 1 failed)
9 scenarios (8 passed, 1 failed)
27 steps (26 passed, 1 failed)

What we do here is, we mock the exception class a bit where we ;

1. Still write the failure message via utilising radish.utils.console_write
2. Do not stop execution, and to do that, do not raise any exception ( Failure )
3. Mark the step as failed via changing the step.state to Step.State.FAILED thus it will be reported as failed on the summary report.

As I changed the code in my local as I explained here above, this works without any problem for terraform-compliance.

Hope I could explain the situation. Please let me know if you need any more details on any part that I couldn't explain it well.

Thanks!

[eerkunt]

I will try to override, mock, patch something for the radish.stepmodel.Step within 0.13.1 till we have an agreement (or implementation) :)

[fliiiix]

@eerkunt can you provide a minimal example for that use-case where i can play around in radish? Lets find a solution for this or close this issue

[fliiiix]

closed feel free to reopen if needed