cli
cli copied to clipboard
terraform-compliance
Possible issue with using `it has` instead of `it contains` with IAM policies
I had previously been using with tf-compliance 1.1.17 the following test to check an IAM role can only be assumed by allowed accounts. This seems to work successfully.
Feature: Reject if the role can be assumed by any account other than specifed
Scenario: Reject if the role can be assumed by ANY account other than specific AWS accounts
Given I have aws_iam_role defined
When it contains assume_role_policy
And it contains Statement
And its Effect is Allow
And its Action is sts:AssumeRole
And it contains Principal
And it contains AWS
Then its value must match the ".*(123456789012|987654321098).*" regex
When I attempt this in 1.2.0, replacing contains with has as prompted:
Feature: Reject if the role can be assumed by any account other than specifed
Scenario: Reject if the role can be assumed by ANY account other than specific AWS accounts
Given I have aws_iam_role defined
When it has assume_role_policy
And it has Statement
And its Effect is Allow
And its Action is sts:AssumeRole
And it has Principal
And it has AWS
Then its value must match the ".*(123456789012|987654321098).*" regex
The test now skips with Can not find any Statement property for aws_iam_role resource in terraform plan.
main.tf: https://pastebin.com/HFwezWjx plan.out.json: https://pastebin.com/kBh0KGdq
Looks like I self-panicked about something completely different :)
Yes this is working as intended. The difference is, on old
When it contains assume_role_policy
It was actually working like a Then function where instead of "filtering", it was "drilling down" into the resource. And on the next step (And it contains Statement) it was using the data that was coming from the previous step. Normally all When functions must not change any data that has been used in the steps.
With a better example, assume that I have 2 resources ;
resource_a
- value_a
- value_b
- value_b_a
- value_b_b
resource_b
- value_a
- value_c
- value_c_a
- value_c_b
When using When it contains value_c, the data for the next step becomes ;
- value_c
- value_c_a
- value_c_b
but actually, it should have been ;
resource_b
- value_a
- value_c
- value_c_a
- value_c_b
So, new When it has value_c does the above, but old When it contains drills down the resource and use that found data for the next step. That should not happen, but it looks like we also need to have a drilling down capability with a SKIP option. Since using Then it must contain instead of When it has will exit if a parameter is not found.
I will have a deeper look onto this one and find a good solution that will help everyone.
I'm seeing something weird, swapping it contains with it has in a feature file that's similar to Tags example:
Feature: Tags should be used to identify resources
Scenario Outline: Ensure that specific tags are defined
Given I have resource that supports tags defined
When it contains tags
Then it must contain <tags>
And its value must match the "<value>" regex
Examples:
| tags | value |
| Name | .+ |
gives success:
% terraform-compliance --with-traceback -f compliance -p plan.out.json
terraform-compliance v1.2.1 initiated
🚩 Features : /target/compliance
🚩 Plan File : /target/plan.out.json
🚩 Running tests. 🎉
Feature: Tags should be used to identify resources # /target/compliance/tags.feature
Scenario Outline: Ensure that specific tags are defined
Given I have resource that supports tags defined
When it contains tags
Then it must contain <tags>
And its value must match the "<value>" regex
Examples:
| tags | value |
❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
| Name | .+ |
1 features (1 passed)
2 scenarios (2 passed)
8 steps (8 passed)
Run 1590597526 finished within a moment
but replacing it contains tags with it has tags results in a failure:
% terraform-compliance --with-traceback -f compliance -p plan.out.json
terraform-compliance v1.2.1 initiated
🚩 Features : /target/compliance
🚩 Plan File : /target/plan.out.json
🚩 Running tests. 🎉
Feature: Tags should be used to identify resources # /target/compliance/tags.feature
Scenario Outline: Ensure that specific tags are defined
Given I have resource that supports tags defined
When it has tags
Then it must contain <tags>
And its value must match the "<value>" regex
Examples:
| tags | value |
Failure: Name property in x resource does not match with .+ case insensitive regex. It is set to .
Failure: Name property in y resource does not match with .+ case insensitive regex. It is set to .
1 features (0 passed, 1 failed)
2 scenarios (0 passed, 1 failed)
8 steps (3 passed, 1 failed)
Run 1590597687 finished within a moment
❗ ERROR: Hook 'console_writer_after_each_scenario' from /usr/local/lib/python3.7/site-packages/radish/extensions/formatters/gherkin.py:425 raised: 'AttributeError: 'NoneType' object has no attribute 'split''
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/radish/hookregistry.py", line 132, in call
func(model, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/radish/extensions/formatters/gherkin.py", line 479, in console_writer_after_each_scenario
for l in failed_step.failure.traceback.split("\n")[:-2]
AttributeError: 'NoneType' object has no attribute 'split'
I don't know if this is related or a separate issue. Unfortunately I can't share the plan, but I can assure you the tags are not empty.
Hello bobdah,
Thanks for reporting this :tada:
The reason why it doesn't work when it is switched (contains -> has) is because When it contains has does not drill down the tags value.
So this test should work ;
Scenario Outline: Ensure that specific tags are defined
Given I have resource that supports tags defined
When it has tags
Then it must contain tags
Then it must contain <tags>
And its value must match the "<value>" regex
Can you have a try ?
Scenario Outline: Ensure that specific tags are defined Given I have resource that supports tags defined When it has tags Then it must contain tags Then it must contain <tags> And its value must match the "<value>" regex
I was having the same issue and this resolved it for me. Thanks @eerkunt !
I'm also having the similar issue that @pontinjx reported, in my case its a s3 bucket policy
The scenario fails to find the Statement inside a policy even though it exists.
Given I have aws_s3_bucket_policy defined
When it has policy
Then it must contain policy
Then it must contain Statement
Failure: aws_s3_bucket_policy.b does not have Statement property
This is weird, I am actively using this on the same use case.
Is it possible to send the plan.out.json if its not confidential ?
Sure @eerkunt please see the attached plan json plan.json.txt
also see the main.tf just in case
resource "aws_s3_bucket" "b" {
bucket = "my_tf_test_bucket"
}
resource "aws_s3_bucket_policy" "b" {
bucket = aws_s3_bucket.b.id
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "MYBUCKETPOLICY",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::my_tf_test_bucket/*",
"Condition": {
"Bool": { "aws:SecureTransport" : false }
}
}
]
}
POLICY
}
so, basically this is what I am trying to do
my original Scenario which was working fine with previous versions (i.e. prior to V 1.2.0)
Test1:
Scenario: aws_s3_bucket_policy test1
Given I have aws_s3_bucket_policy defined
When it contains policy
And it contains Statement
And its Effect is Deny
Then it must contain Condition
And it must contain Bool
And it must contain aws:SecureTransport
And its value must be false
Result:
since I got the Warnings to convert When it contains step to When it has, I started with the below scenario, however it failed
Test2:
Scenario: aws_s3_bucket_policy test2
Given I have aws_s3_bucket_policy defined
When it has policy
Then it must contain policy
Then it must contain Statement
Results: Its not able to find the Statement inside policy even though it exists
Please let me know how can I convert my whole Scenario (test1) to use When it has step?
another questions:
Do we support And step after Then as shown below?
Then it must contain key
And its key is value
please let me know... thank in advance!
Hi @eerkunt ,
I am wondering if you have had a chance to check my above issue? thanks in advance!
I was also having the same original problem when I changed "contains" => "has". I haven't tested it yet but it seems like @eerkunt approach would work for one of my scenarios where there is a single "Where" clause:
Scenario Outline: Ensure that my specific tags are defined
Given I have resource that supports tags defined
When it contains tags
Then it must contain <tag_keys>
And its value must match the "<pattern>" regex
Examples:
| tag_keys | pattern |
| application | ^([a-z]+[0-9]*)*[\-]?([a-z0-9]+[\-])*[a-z0-9]+$
Not sure how it would work for the following (which I am using to check optional tags):
Scenario: Verify value if cost_center tag is defined
Given I have resource that supports tags defined
When it contains tags
When it contains cost_center
Then its value must match the "^([A-Za-z0-9]+)$" regex
This is what I am trying to do in the scenario:
- check that it contains a tags object (and get the tags object)
- then check if the tags object contains a cost_center tag/key
- If both are true, then check the value of cost_center against the regex
Following the approach @eerkunt mentioned, would it be as follows?:
Scenario: Verify value if cost_center tag is defined
Given I have resource that supports tags defined
When it has tags
And it has cost_center
Then it must contain tags
Then it must contain cost_center
Then its value must match the "^([A-Za-z0-9]+)$" regex
Please let me know.
I was also having the same original problem when I changed "contains" => "has". I haven't tested it yet but it seems like @eerkunt approach would work for one of my scenarios where there is a single "Where" clause:
Scenario Outline: Ensure that my specific tags are defined
Given I have resource that supports tags defined
When it contains tags
Then it must contain <tag_keys>
And its value must match the "<pattern>" regex
Examples:
| tag_keys | pattern |
| application | ^([a-z]+[0-9]*)*[\-]?([a-z0-9]+[\-])*[a-z0-9]+$
Not sure how it would work for the following (which I am using to check optional tags):
Scenario: Verify value if cost_center tag is defined
Given I have resource that supports tags defined
When it contains tags
When it contains cost_center
Then its value must match the "^([A-Za-z0-9]+)$" regex
This is what I am trying to do in the scenario:
- check that it contains a tags object (and get the tags object)
- then check if the tags object contains a cost_center tag/key
- If both are true, then check the value of cost_center against the regex
Following the approach @eerkunt mentioned, would it be as follows?:
Scenario: Verify value if cost_center tag is defined
Given I have resource that supports tags defined
When it has tags
And it has cost_center
Then it must contain tags
Then it must contain cost_center
Then its value must match the "^([A-Za-z0-9]+)$" regex
Please let me know.
@anthonycolon25
And it has cost_center might not find cost_center as it's buried under tags. I would swap the order of steps to
Scenario: Verify value if cost_center tag is defined
Given I have resource that supports tags defined
When it has tags
Then it must contain tags
When it has cost_center
Then it must contain cost_center
Then its value must match the "^([A-Za-z0-9]+)$" regex
This way, you will be drilling down to tags before looking for cost_center.
Hi @vrbcntrl
I tried the following Scenarios on 1.2.10 and all seem to pass:
Scenario: aws_s3_bucket_policy test1
Given I have aws_s3_bucket_policy defined
When it contains policy
And it contains Statement
And its Effect is Deny
Then it must contain Condition
And it must contain Bool
And it must contain aws:SecureTransport
And its value must be false
Scenario: aws_s3_bucket_policy test2
Given I have aws_s3_bucket_policy defined
When it has policy
Then it must contain policy
Then it must contain Statement
Scenario: aws_s3_bucket_policy test1 but with has
Given I have aws_s3_bucket_policy defined
When it has policy
Then it must contain policy
Then it must contain Statement
When its Effect is Deny
Then it must contain Condition
And it must contain Bool
And it must contain aws:SecureTransport
And its value must be false
AND is treated as THEN or WHEN depending on whichever was used in the previous step. Since there are no Then its key is value steps defined as of 1.2.10, you would get a syntax error using
Then it must contain key
And its key is value
I would abstain from using When it contains as it might not be supported in the future.
Please let me know if this doesn't work!
Looks like I self-panicked about something completely different :)
Yes this is working as intended. The difference is, on old
When it contains assume_role_policyIt was actually working like a
Thenfunction where instead of "filtering", it was "drilling down" into the resource. And on the next step (And it contains Statement) it was using the data that was coming from the previous step. Normally allWhenfunctions must not change any data that has been used in the steps.With a better example, assume that I have 2 resources ;
resource_a - value_a - value_b - value_b_a - value_b_b resource_b - value_a - value_c - value_c_a - value_c_bWhen using
When it contains value_c, the data for the next step becomes ;- value_c - value_c_a - value_c_bbut actually, it should have been ;
resource_b - value_a - value_c - value_c_a - value_c_bSo, new
When it has value_cdoes the above, but oldWhen it containsdrills down the resource and use that found data for the next step. That should not happen, but it looks like we also need to have a drilling down capability with aSKIPoption. Since usingThen it must containinstead ofWhen it haswill exit if a parameter is not found.I will have a deeper look onto this one and find a good solution that will help everyone.
Hello @eerkunt , I did something like this,
Scenario: Reject if the role can be assumed by ANY service other than specific AWS service
Given I have aws_iam_role defined
When it has assume_role_policy
And it contains Statement
And its Effect is Allow
And its Action is sts:AssumeRole
And it has Principal
And it contains Service
Then its value must match the "<value>" regex
This does not skip the test case, But it shows the deprecation warnings as below.
Scenario: Reject if the role can be assumed by ANY service other than specific AWS service
Given I have aws_iam_role defined
When it has assume_role_policy
❗ WARNING: "When it contains Statement" step functionality will be changed on future versions and the functionality will be same as "When it has Statement" step. Please use the latter.
And it contains Statement
And its Effect is Allow
And its Action is sts:AssumeRole
And it has Principal
❗ WARNING: "When it contains Service" step functionality will be changed on future versions and the functionality will be same as "When it has Service" step. Please use the latter.
And it contains Service
Then its value must match the "<value>" regex
Is there any way I can avoid these warnings and still able to run test cases?
Looks like I self-panicked about something completely different :)
Yes this is working as intended. The difference is, on old
When it contains assume_role_policyIt was actually working like a
Thenfunction where instead of "filtering", it was "drilling down" into the resource. And on the next step (And it contains Statement) it was using the data that was coming from the previous step. Normally allWhenfunctions must not change any data that has been used in the steps.With a better example, assume that I have 2 resources ;
resource_a - value_a - value_b - value_b_a - value_b_b resource_b - value_a - value_c - value_c_a - value_c_bWhen using
When it contains value_c, the data for the next step becomes ;- value_c - value_c_a - value_c_bbut actually, it should have been ;
resource_b - value_a - value_c - value_c_a - value_c_bSo, new
When it has value_cdoes the above, but oldWhen it containsdrills down the resource and use that found data for the next step. That should not happen, but it looks like we also need to have a drilling down capability with aSKIPoption. Since usingThen it must containinstead ofWhen it haswill exit if a parameter is not found.I will have a deeper look onto this one and find a good solution that will help everyone.
@eerkunt Any Luck on the updating to has over contains. I also switched over to trying to use has on the examples on the terraform-compliance website because of the deprecation warning and my tests did break due to not having the drill down like contains does. If anything maybe dual examples of old contains vs new has with drilldowns/selectors posted on examples pages
@Kudbettin Thank you for suggesting the multiple When-Then approach, it solves my problem.
On the other hand: "If you mix the keywords in a some interesting way, you are possible destroying one of the important properties of Gherkin, the ability to function as a living documentation. It is possible to create a scenario that is unreadable this way." StackOverflow.
@eerkunt Please describe the reason behind the plan to obsolete the drilling down When feature. From my perspective, this behaviour was useful and readable.
