cli icon indicating copy to clipboard operation
cli copied to clipboard

Bug for `when it has` vs `when it contains` can create confusion due to stash differences, MORE context warning requested

Open mdesmarest opened this issue 3 years ago • 1 comments

Description : Using Version 1.3.13 The warning message WARNING: "When it contains bucket" step functionality will be changed on future versions and the functionality will be same as "When it has bucket" step. Please use the latter. creates an issue because they are not the same. When reading the docs, literally referencing using when it has, you can make this determination, however it is very easy to confused and replace one for one and create confusion.

When it contains drills down, when it has does not and requires an additional Then to drill down

      When it has bucket
      Then it must have bucket

To Reproduce When using contains this works as intended:

Scenario: Check for the presence of a new aws_s3_bucket
        Given I have aws_s3_bucket defined
>> Press enter to continue
        When its actions metadata has create
>> Press enter to continue
	❗ WARNING: "When it contains bucket" step functionality will be changed on future versions and the functionality will be same as "When it has bucket" step. Please use the latter.
        When it contains bucket
>> Press enter to continues
[
    {
        "address": "aws_s3_bucket.badname1",
        "values": "my-tf-test-bucket_44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "aws_s3_bucket.badname2",
        "values": "Ny-tf-test-bucket44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "aws_s3_bucket.goodname",
        "values": "my-tf-test-bucket44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "module.moduletest.aws_s3_bucket.backup",
        "values": "funky-spraynpray-817408693477",
        "type": "aws_s3_bucket"
    },
    {
        "address": "module.moduletest.aws_s3_bucket.main",
        "values": "funky-spraynpray",
        "type": "aws_s3_bucket"
    }
]
>> Press enter to continue
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket_44444.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Ny-tf-test-bucket44444.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket44444.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to funky-spraynpray.
        Then its value must be "funky-spraynpray-817408693477"
          Failure: 

When using When it has you can see that the stash is populated with all of the data, rather than drilled down to the bucket property. So if you continue with below:

    Scenario: Check for the presence of a new aws_s3_bucket
        Given I have aws_s3_bucket defined
>> Press enter to continue
        When its actions metadata has create
>> Press enter to continue
        When it has bucket
>> Press enter to continues
[
    {
        "address": "aws_s3_bucket.badname1",
        "mode": "managed",
        "type": "aws_s3_bucket",
        "name": "badname1",
        "provider_name": "registry.terraform.io/hashicorp/aws",
        "values": {
            "acl": "private",
            "bucket": "my-tf-test-bucket_44444",
            "bucket_prefix": null,
            "cors_rule": [],
            "force_destroy": false,
            "grant": [],
            "lifecycle_rule": [],
            "logging": [],
            "object_lock_configuration": [],
            "policy": null,
            "replication_configuration": [],
            "server_side_encryption_configuration": [],
            "tags": {
                "Environment": "Dev",
                "Name": "My bucket2"
            },
            "website": []
        },
        "actions": [
            "create"
        ]
    },
    {
        "address": "aws_s3_bucket.badname2",
        "mode": "managed",
        "type": "aws_s3_bucket",
        "name": "badname2",
        "provider_name": "registry.terraform.io/hashicorp/aws",
        "values": {
            "acl": "private",
            "bucket": "Ny-tf-test-bucket44444",
            "bucket_prefix": null,
            "cors_rule": [],
            "force_destroy": false,
            "grant": [],
            "lifecycle_rule": [],
            "logging": [],
            "object_lock_configuration": [],
            "policy": null,
            "replication_configuration": [],
            "server_side_encryption_configuration": [],
            "tags": {
                "Environment": "Dev",
                "Name": "My bucket3"
            },
            "website": []
        },
        "actions": [
            "create"
        ]
    },
****FURTHER DATA REDACTED*******

	Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket_44444.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Dev.
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to My bucket2.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Ny-tf-test-bucket44444.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Dev.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to My bucket3.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket44444.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Dev.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to My bucket1.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to AES256.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yolo.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yeah.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to ok.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to True.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to funky-spraynpray.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to .
		Failure: bucket property in module.moduletest.aws_s3_bucket.backup resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to REDACTED

		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to .
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Enabled.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to AES256.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yolo.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yeah.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to ok.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to True.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to private.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to None.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to AES256.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yolo.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to yeah.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to ok.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to True.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to False.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to True.
        Then its value must be "funky-spraynpray-817408693477"

You fail incorrectly, based on the assumption when When it has is a 1 to 1 replacement for When it contains

When using When it has something you must accompany it with a Then it must have something to accomplish what When it contains used to provide. See below:

 Scenario: Check for the presence of a new aws_s3_bucket
        Given I have aws_s3_bucket defined
>> Press enter to continue
        When its actions metadata has create
>> Press enter to continue
        When it has bucket
>> Press enter to continue
        Then it must have bucket
>> Press enter to continues
[
    {
        "address": "aws_s3_bucket.badname1",
        "values": "my-tf-test-bucket_44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "aws_s3_bucket.badname2",
        "values": "Ny-tf-test-bucket44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "aws_s3_bucket.goodname",
        "values": "my-tf-test-bucket44444",
        "type": "aws_s3_bucket"
    },
    {
        "address": "module.moduletest.aws_s3_bucket.backup",
        "values": "funky-spraynpray-817408693477",
        "type": "aws_s3_bucket"
    },
    {
        "address": "module.moduletest.aws_s3_bucket.main",
        "values": "funky-spraynpray",
        "type": "aws_s3_bucket"
    }
]
>> Press enter to continue
		Failure: bucket property in aws_s3_bucket.badname1 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket_44444.
		Failure: bucket property in aws_s3_bucket.badname2 resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to Ny-tf-test-bucket44444.
		Failure: bucket property in aws_s3_bucket.goodname resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to my-tf-test-bucket44444.
		Failure: bucket property in module.moduletest.aws_s3_bucket.main resource does not match with ^funky\-spraynpray\-817408693477$ case insensitive regex. It is set to funky-spraynpray.
        Then its value must be "funky-spraynpray-817408693477"
          Failure: 

Expected behavior : I would advise replacing WARNING: "When it contains bucket" step functionality will be changed on future versions and the functionality will be same as "When it has bucket" step. Please use the latter.

with WARNING: "When it contains bucket" step functionality will be changed on future versions and the functionality will be same as "When it has bucket" step. To replicate then same functionality please use "When it has bucket" and add "Then it must have bucket" to drill into the stash properly

Tested versions :

  • <terraform-compliance version (terraform-compliance -v)>1.3.13
  • <terraform version (terraform -v)>
  • <python runtime version, if running as a python package (python --version)>

Additional context Add any other context about the problem here.

mdesmarest avatar Mar 08 '21 18:03 mdesmarest

Thanks for pointing this out @mdesmarest . This has been raised before, it looks like its creating more problems than fixing. Will produce a document about this and provide a link to that.

Also provide a parameter to the cli to surpress WARNING messages in case you are already aware of this problem.

eerkunt avatar Mar 09 '21 10:03 eerkunt