Potentially incorrect docs for synthetics test resource

[brandonmwest]

There are two instances in https://github.com/DataDog/terraform-provider-datadog/edit/master/docs/resources/synthetics_test.md#L321 where is_critical is described as "Determines whether or not to consider the entire test as failed if this step fails. Can be used only if allow_failure is true."

As allow_failure "Determines whether or not to continue with test if this step fails," it seems that is_critical should instead require that allow_failure be set to false. If the test does not complete, is_critical cannot fail the entire test.

This has been reported by a user on slack who also states that existing tests that are imported do actually have is_critical set to true while allow_failure is false.

[nkzou]

Hi, this is the intended behavior, although the docs should probably be reworded to be more clear - by default, a failure in any step means that the entire test fails and execution of any remaining steps is stopped. allow_failure doesn't refer to the pass/fail status of the test, but the process of running it, telling the synthetics test to continue with the following steps regardless of that step's failure.is_critical refers to pass/fail status, telling synthetics to consider the whole test a failure despite allow_failure letting subsequent steps continue running.

Some examples (assuming all other steps are passing): If allow_failure is true and is_critical is false, the test will still be considered passing and the rest of the steps will run as normal. If allow_failure is true and is_critical is true, the test will be considered failing but the remaining steps will run as normal. Finally, if allow_failure is false, the test fails and no additional steps will run. This is default behavior

Hopefully that made sense - we'll see what we can do about the docs to minimize the confusion.

[brandonmwest]

Thank you, makes sense!