ExpectError supports for Regexp is not sufficient in some cases

[obourdon]

SDK version

2.19.0

Use-cases

See details in https://github.com/hashicorp/terraform-provider-aws/issues/25986

in case of early fatal AWS ASGs errors occurring after timeout, the error message is exactly the same as reported with the fix provided plus a prefix

Waiting up to .*: Need at least 1 healthy instances in ASG(?ms).*SAME_ERROR_MESSAGE

but due to the fact that the basic Golang regex implementation does not support Perl like regex and more specifically exclude pattern, I did not find any proper way using the current SDK code to provide tests which fail under timeout and which succeed with early error reporting

Attempted Solutions

See above

Proposal

See https://github.com/hashicorp/terraform-plugin-sdk/pull/1015

References

• https://github.com/hashicorp/terraform-provider-aws/issues/25986
• https://github.com/hashicorp/terraform-provider-aws/pull/25987

[bflad]

Hi @obourdon 👋 Thank you for raising this.

For more advanced error checking/handling, there is the resource.TestCase type ErrorCheck field, which accepts a func(error) error that can be used to skip tests when encountering certain errors in test steps, or in your particular use case, use custom logic to detect certain errors and return nil so the detected error causes the step to pass while other errors will still cause the step to fail.

Can you please try that in your situation?

[obourdon]

@bflad many thanks for your comments. I indeed considered using this but it did not seem to fit what I was trying to achieve because I don't want the test to be skipped if the condition is met or not I really want to make the difference between getting:

- <SOME-COMMON-SUFFIX-REGEXP>
- <RESOURCE-TIMEOUT-CASE-ADDED-PREFIX>(?ms).*<SOME-COMMON-SUFFIX-REGEXP>

to see that either my test and fix fails/succeed

Unless I misunderstood your point, I don't think ErrorCheck field allows to achieve this and, of course, ExpectedError neither due to Golang regexp "restrictions"

[obourdon]

@bflad any progress on this ?

[bflad]

I apologize that I explicitly mentioned skipping in my previous response, but that is just one action the generic ErrorCheck functionality can be used for in acceptance testing (and one that the AWS provider codebase uses extensively to handle cross-region differences). It can also be used to rewrite errors or omit them completely.

For example with a completely inline unit test:

func Test1014(t *testing.T) {
	resource.UnitTest(t, resource.TestCase{
		ErrorCheck: func(err error) error {
			if err == nil {
				return nil
			}
			if strings.Contains(err.Error(), "example prefix: ") {
				return err
			}
			if strings.Contains(err.Error(), "example error") {
				return nil
			}
			return err
		},
		Steps: []resource.TestStep{
			{
				Config: `resource "example_resource" "test" {}`,
				ProviderFactories: map[string]func() (*schema.Provider, error){
					"example": func() (*schema.Provider, error) {
						return &schema.Provider{
							ResourcesMap: map[string]*schema.Resource{
								"example_resource": {
									CreateContext: func(_ context.Context, d *schema.ResourceData, _ interface{}) diag.Diagnostics {
										return diag.Errorf("example prefix: example error")
									},
									DeleteContext: func(_ context.Context, _ *schema.ResourceData, _ interface{}) diag.Diagnostics {
										return nil
									},
									ReadContext: func(_ context.Context, _ *schema.ResourceData, _ interface{}) diag.Diagnostics {
										return nil
									},
									Schema: map[string]*schema.Schema{
										"id": {
											Computed: true,
											Type:     schema.TypeString,
										},
									},
								},
							},
						}, nil
					},
				},
			},
		},
	})
}

Appropriately returns the error with the unwanted prefix:

--- FAIL: Test1014 (0.39s)
    /Users/bflad/src/github.com/hashicorp/terraform-plugin-sdk/helper/resource/teststep_providers_test.go:229: Step 1/1 error: Error running apply: exit status 1
        
        Error: example prefix: example error
        
          with example_resource.test,
          on terraform_plugin_test.tf line 1, in resource "example_resource" "test":
           1: resource "example_resource" "test" {}

Updating the error to just return diag.Errorf("example error"), the test succeeds.

[obourdon]

Sorry I did not had a look back at this earlier on I'll give your solution a try soon. Thanks for the details

[bflad]

Hi @obourdon 👋 Since we haven't heard anything back, I'm going to close this issue for now.

With the introduction of the terraform-plugin-testing module (migration guide), we will be handling feature requests relating to the helper/resource "testing framework" in the terraform-plugin-sdk repository over to the new terraform-plugin-testing repository going forward, if you would like to create a new followup issue over there.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.