Getting exceptions with tfparse 0.6.14

[JonZeolla]

This issue is new as of 0.6.13; I think it's related to the trivy update from v0.50.1 to v0.58.0 in this PR but I can't prove it yet.

Error

goroutine 17 [running, locked to thread]: github.com/zclconf/go-cty/cty.Value.AsString({{{0x7f474917ba70?, 0xc0001141e1?}}, {0x0?, 0x0?}}) /tmp/tmpc9zo0uqm/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1390 +0x10b github.com/aquasecurity/trivy/pkg/iac/terraform.postProcessValues(0xc000c7c380, 0xc000d81860) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/terraform/presets.go:52 +0x393 github.com/aquasecurity/trivy/pkg/iac/terraform.(*Block).Values(0xc000c7c380) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/terraform/block.go:580 +0x185 github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).getResources(0xc000818280) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:581 +0x18e github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateStep(0xc000818280) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:99 +0x17b github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSteps(0xc000818280) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:246 +0x152 github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).EvaluateAll(0xc000818280, {0x7f474917b680, 0x7f4749fbb280}) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:135 +0x1eb github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSubmodule(0xc000900a00, {0x7f474917b680, 0x7f4749fbb280}, 0xc000953740) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:228 +0x1bb github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSubmodules(0xc000900a00, {0x7f474917b680, 0x7f4749fbb280}, 0xc00092ca50) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:164 +0x43f github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).EvaluateAll(0xc000900a00, {0x7f474917b680, 0x7f4749fbb280}) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:142 +0x294 github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSubmodule(0xc0005c08c0, {0x7f474917b680, 0x7f4749fbb280}, 0xc000928300) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:228 +0x1bb github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSubmodules(0xc0005c08c0, {0x7f474917b680, 0x7f4749fbb280}, 0xc0005ae570) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:164 +0x43f github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).EvaluateAll(0xc0005c08c0, {0x7f474917b680, 0x7f4749fbb280}) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:142 +0x294 github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*Parser).EvaluateAll(0xc0004666c0, {0x7f474917b680, 0x7f4749fbb280}) /tmp/tmpc9zo0uqm/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/parser.go:342 +0x90 github.com/cloud-custodian/tfparse/gotfparse/pkg/converter.NewTerraformConverter({0xc00006c0c0, 0x59}, {0xc000622140, 0x2, 0x7f4748e46b60?}) /tmp/tmpc9zo0uqm/src/github.com/cloud-custodian/tfparse/gotfparse/gotfparse/pkg/converter/converter.go:372 +0x187 main.Parse(0x0?, 0x0, 0x0, 0x0, 0x1a79a90, 0x0, 0x7fffd9ec1530) /tmp/tmpc9zo0uqm/src/github.com/cloud-custodian/tfparse/gotfparse/gotfparse/cmd/tfparse/main.go:46 +0x5bc

[ajkerrigan]

Thanks for the report - would you be able to share the Terraform module you're scanning when you hit this error? Following the stack trace there it looks like it might be dying while trying to patch the arn of an aws_s3_bucket block here.

But using tfparse 0.6.13 with a test module that has s3 buckets defined, I'm seeing the arn outputs I expect. Wonder if there's a specific type of bucket name that's causing it trouble, or if I'm looking in the wrong spot :monocle_face: .

[JonZeolla]

Thanks for taking a look. I will look into this a bit more. Unfortunately it's in one of my more sensitive and more complex modules which use many other modules. I did try to get a MRE but haven't had a chance to nail it down yet.

[JonZeolla]

My problem appears to have been fixed in the latest release. I couldn't point to anything specific in the list of changes so my guess is it's some unpinned dependency? Either way, closing the issue.

[JonZeolla]

Unfortunately my testing was incomplete and I am still having this issue. I will look into getting you a MRE for reproducing

[JonZeolla]

So, if you clone https://github.com/primeharbor/org-kickstart and checkout 839443dbeb0460de718dbd46ecc3923a58c74db7, then run the following you can reproduce the issue like this:

➜ uv init
Initialized project `org-kickstart`
 org-kickstart on  remotes/origin/HEAD
➜ uv add tfparse
Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 4 packages in 192ms
Prepared 3 packages in 498ms
Installed 3 packages in 2ms
 + cffi==1.17.1
 + pycparser==2.22
 + tfparse==0.6.14
 org-kickstart on  remotes/origin/HEAD [?]
➜ uv run python3
Python 3.11.10 (main, Oct  7 2024, 23:25:27) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tfparse import load_from_path
>>> from pathlib import Path
>>> load_from_path(Path("."))
panic: value is null

goroutine 17 [running, locked to thread]:
github.com/zclconf/go-cty/cty.Value.AsString({{{0x10a783c90?, 0x14000010251?}}, {0x0?, 0x0?}})
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1390 +0x114
github.com/aquasecurity/trivy/pkg/iac/terraform.postProcessValues(0x14000703dc0, 0x14000b694d0)
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/terraform/presets.go:52 +0x39c
github.com/aquasecurity/trivy/pkg/iac/terraform.(*Block).Values(0x14000703dc0)
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/terraform/block.go:580 +0x174
github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).getResources(0x140001775e0)
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:581 +0x178
github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateStep(0x140001775e0)
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:99 +0x150
github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).evaluateSteps(0x140001775e0)
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:246 +0x108
github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*evaluator).EvaluateAll(0x140001775e0, {0x10a7838a0, 0x10b5d9380})
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/evaluator.go:135 +0x16c
github.com/aquasecurity/trivy/pkg/iac/scanners/terraform/parser.(*Parser).EvaluateAll(0x140006be120, {0x10a7838a0, 0x10b5d9380})
        /var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/pkg/mod/github.com/cloud-custodian/[email protected]/pkg/iac/scanners/terraform/parser/parser.go:342 +0x78
github.com/cloud-custodian/tfparse/gotfparse/pkg/converter.NewTerraformConverter({0x1400062c0c0, 0x1}, {0x14000612000, 0x2, 0x10a44eb20?})
        /private/var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/src/github.com/cloud-custodian/tfparse/gotfparse/gotfparse/pkg/converter/converter.go:372 +0x190
main.Parse(0x0?, 0x0, 0x0, 0x0, 0x6000017d8070, 0x0, 0x16b709ee0)
        /private/var/folders/0j/bwqcs4y508s2n4ck4dhf3rpc0000gn/T/tmp6t3y9h_a/src/github.com/cloud-custodian/tfparse/gotfparse/gotfparse/cmd/tfparse/main.go:46 +0x4e8

[JonZeolla]

@ajkerrigan, after some digging this may be due to https://github.com/aquasecurity/trivy/issues/8093 which was fixed in trivy v0.59.0.

I saw that tfparse vendors trivy and has an open PR for this upgrade in https://github.com/cloud-custodian/trivy/pull/8 but when attempting to use that new version in https://github.com/cloud-custodian/tfparse/pull/232 there were some test issues. I wonder if v0.59.1 or v0.60.0 would work better?

I saw @kapilt's note about c7n-left unit tests, but in poking around I only found references to that outside of main, here, and here. I'm guessing the need is to add it to the latter?

I would like to contribute to a fix here if it's at all possible - how can I help?

[ajkerrigan]

Thanks for the repro and digging @JonZeolla - I had been meaning to revisit a trivy update anyway so that's a super helpful test. As far as I can tell updating to trivy v0.60.0 does fix at least the repro you shared. I was seeing the same exception locally using tfparse 0.6.14, but testing the same steps with #239 looked clean.

If you have other non-public code that was hitting this issue, it would be useful to see if #239 fixes those cases too.

[JonZeolla]

My initial testing looks good 👍

I will test more completely after the release as well. TY!

[JonZeolla]

This issue is confirmed as fixed on 0.6.15