[BUG] Validation fails on CFT with rules created in rulegen v2.0.2/v2.0.3

[Relativity11]

Describe the bug Validating some CloudFormation Templates against the rules created from them with rulegen fail ambiguously and do not give good debugging to describe why a failure occurred.

To Reproduce Please supply:

1. Run a cfn-guard rulegen against a complex CFT.
2. Run a cfn-guard validate against the complex CFT and the generated rules.

Snippet Example: let aws_route53_recordset_resources = Resources.*[ Type == 'AWS::Route53::RecordSet' ] rule aws_route53_recordset when %aws_route53_recordset_resources !empty { %aws_route53_recordset_resources.Properties.Type == "A" %aws_route53_recordset_resources.Properties.Name IN [["",["SubdomainMaster",".","HostedZoneName"]], ["",["SubdomainDefault",".","HostedZoneName"]], ["",["SubdomainInternal",".","HostedZoneName"]], ["",["SubdomainWild",".","HostedZoneName"]]] %aws_route53_recordset_resources.Properties.TTL == "900" %aws_route53_recordset_resources.Properties.Comment == "DNS name for my instance." %aws_route53_recordset_resources.Properties.HostedZoneName == "HostedZoneName" %aws_route53_recordset_resources.Properties.ResourceRecords IN [["Infra1.PrivateIp"], ["Master.PrivateIp"]] }

  MasterRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName: !Ref 'HostedZoneName'
      Comment: DNS name for my instance.
      Name: !Join ['', [!Ref 'SubdomainMaster', ., !Ref 'HostedZoneName']]
      Type: A
      TTL: '900'
      ResourceRecords:
      - !GetAtt Master.PrivateIp
  InternalRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName: !Ref 'HostedZoneName'
      Comment: DNS name for my instance.
      Name: !Join ['', [!Ref 'SubdomainInternal', ., !Ref 'HostedZoneName']]
      Type: A
      TTL: '900'
      ResourceRecords:
      - !GetAtt Master.PrivateIp
  SubdomainRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName: !Ref 'HostedZoneName'
      Comment: DNS name for my instance.
      Name: !Join ['', [!Ref 'SubdomainDefault', ., !Ref 'HostedZoneName']]
      Type: A
      TTL: '900'
      ResourceRecords:
      - !GetAtt Infra1.PrivateIp
  WildcardRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName: !Ref 'HostedZoneName'
      Comment: DNS name for my instance.
      Name: !Join ['', [!Ref 'SubdomainWild', ., !Ref 'HostedZoneName']]
      Type: A
      TTL: '900'
      ResourceRecords:
      - !GetAtt Infra1.PrivateIp

Expected behavior All areas "PASS" when running rules against their own CFT.

Screenshots

Operating System: MacOS 10.14 Fedora 34

Additional context Some CFT fully PASS when running validation against their own rules, some do not. Shouldn't validation ensure a full PASS when running CFT against the rules created in rulegen against themselves - always?

We have many templates and are unable to prep them individually for CFN-Guard. Attempted multiple branches to see if failure persists.

Thank you for any input!

[dchakrav-github]

Thanks for report @Relativity11.

1. Can you attach the sample CFN template here used for rulegen?
2. Can you provide a sample run with --show-clause-failures?
3. Can you provide the output of a verbose run using the -v flag for validate?

--show-clause-failures is the one that shows what comparisons failed. It seems related to the issue that current rust language bindings when reading in tags like !Ref gets dropped when mapping to a value. Appreciate inputs for the above

[dchakrav-github]

Recovered snippet as

Resources:
    MasterRecord: 
        Type: AWS::Route53::RecordSet 
        Properties: 
            HostedZoneName: !Ref 'HostedZoneName' 
            Comment: DNS name for my instance. 
            Name: !Join ['', [!Ref 'SubdomainMaster', ., !Ref 'HostedZoneName']] 
            Type: A 
            TTL: '900' 
            ResourceRecords: 
                - !GetAtt Master.PrivateIp 
    InternalRecord: 
        Type: AWS::Route53::RecordSet 
        Properties: 
            HostedZoneName: !Ref 'HostedZoneName' 
            Comment: DNS name for my instance. 
            Name: !Join ['', [!Ref 'SubdomainInternal', ., !Ref 'HostedZoneName']] 
            Type: A 
            TTL: '900' 
            ResourceRecords: 
                - !GetAtt Master.PrivateIp 
    SubdomainRecord: 
        Type: AWS::Route53::RecordSet 
        Properties: 
            HostedZoneName: !Ref 'HostedZoneName' 
            Comment: DNS name for my instance. 
            Name: !Join ['', [!Ref 'SubdomainDefault', ., !Ref 'HostedZoneName']] 
            Type: A 
            TTL: '900' 
            ResourceRecords: 
                - !GetAtt Infra1.PrivateIp 
    WildcardRecord: 
        Type: AWS::Route53::RecordSet 
        Properties: 
            HostedZoneName: !Ref 'HostedZoneName' 
            Comment: DNS name for my instance. 
            Name: !Join ['', [!Ref 'SubdomainWild', ., !Ref 'HostedZoneName']] 
            Type: A 
            TTL: '900' 
            ResourceRecords: 
                - !GetAtt Infra1.PrivateIp

Reproduced bug reported with above template. Fix issue and tested it locally. Update being pushed as a part of PR #159

[Relativity11]

Guard team, Really happy to see 2.0.3 and its maturity. Currently the program is working much better with premade templates we have in some of our environment.

However, a few are still giving me issues with the SecurityGroup portion when running against themselves failing at finding a key or array:

CFNGuard 2.0 jacob$ cfn-guard validate -d EC2-R53-SG-KP.yaml -r EC2-R53-SG-KP.rules
EC2-R53-SG-KP.yaml Status = FAIL
PASS rules
EC2-R53-SG-KP.rules/aws_ec2_securitygroupingress    PASS
EC2-R53-SG-KP.rules/aws_route53_recordset           PASS
EC2-R53-SG-KP.rules/aws_route53_hostedzone          PASS
EC2-R53-SG-KP.rules/aws_ec2_instance                PASS
FAILED rules
EC2-R53-SG-KP.rules/aws_ec2_securitygroup           FAIL
---
Evaluation of rules EC2-R53-SG-KP.rules against data EC2-R53-SG-KP.yaml
--
Property traversed until [/Resources/OpInternalSecurityGroup/Properties] in data [EC2-R53-SG-KP.yaml] is not compliant with [EC2-R53-SG-KP.rules/aws_ec2_securitygroup] due to retrieval error. Error Message [Attempting to retrieve array index or key from map at path = /Resources/OpInternalSecurityGroup/Properties , Type was not an array/object map, Remaining Query = SecurityGroupIngress]
--
Rule [EC2-R53-SG-KP.rules/aws_route53_recordset] is compliant for template [EC2-R53-SG-KP.yaml]
Rule [EC2-R53-SG-KP.rules/aws_ec2_securitygroupingress] is compliant for template [EC2-R53-SG-KP.yaml]
Rule [EC2-R53-SG-KP.rules/aws_route53_hostedzone] is compliant for template [EC2-R53-SG-KP.yaml]
Rule [EC2-R53-SG-KP.rules/aws_ec2_instance] is compliant for template [EC2-R53-SG-KP.yaml]
--

Any more detail on what it is failing at finding? A lot of detail is referenced !Ref to others.

[shreyasdamle]

Hi @Relativity11 can you please provide a sample template and rule files for this issue?

[razcloud]

@Relativity11 Reviving this Issue, can you please upload the template and ruleset that you used? Thank you.

[joshfried-aws]

Hi @Relativity11 I am going to go ahead and close out this issue do to inactivity. Feel free to reopen it and provide the clarifications @razcloud had asked for if need be.

Thanks,