find_unused fails with '_pyjq.ScriptRuntimeError: Cannot iterate over null (null)'

[dhirensham]

I've tried this in both linux (WSL2 under Win10 in case it makes any difference) as well as Docker with the same results. The collect command runs successfully. I then run the find_unused command which fails with the following stack trace:

(venv) me@me:/mnt/c/Projects/cloudmapper$ python cloudmapper.py find_unused --accounts myaccount
Traceback (most recent call last):
  File "cloudmapper.py", line 72, in <module>
    main()
  File "cloudmapper.py", line 66, in main
    commands[command].run(arguments)
  File "/mnt/c/Projects/cloudmapper/commands/find_unused.py", line 13, in run
    unused_resources = find_unused_resources(accounts)
  File "/mnt/c/Projects/cloudmapper/shared/find_unused.py", line 115, in find_unused_resources
    find_unused_security_groups(region),
  File "/mnt/c/Projects/cloudmapper/shared/find_unused.py", line 21, in find_unused_security_groups
    for sg in pyjq.all(".SecurityGroups[]", defined_sgs):
  File "/mnt/c/Projects/cloudmapper/venv/lib/python3.8/site-packages/pyjq.py", line 49, in all
    return compile(script, vars, library_paths).all(_get_value(value, url, opener))
  File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561)
_pyjq.ScriptRuntimeError: Cannot iterate over null (null)

I get the same error and stack trace from the Docker instance as well. I did see from #617 that the find_unused command is barely used, but the main reason I've been looking into this sort of application is to help identify unused resources from a cost management perspective rather than security audit. So for me, finding orphaned resources that don't link back to running instances is the primary goal.

NB: the collect command does return some errors... it doesn't look like those are related to the find_unused issue at all, but including them just in case:

Summary: 1415 APIs called. 4 errors
Failures:
  s3.get_bucket_acl({'Bucket': 'elasticbeanstalk-af-south-1-988217567069'}): An error occurred (IllegalLocationConstraintException) when calling the GetBucketAcl operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
  s3.get_bucket_policy({'Bucket': 'elasticbeanstalk-af-south-1-988217567069'}): An error occurred (IllegalLocationConstraintException) when calling the GetBucketPolicy operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
  s3.get_bucket_logging({'Bucket': 'elasticbeanstalk-af-south-1-988217567069'}): An error occurred (IllegalLocationConstraintException) when calling the GetBucketLogging operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
  s3.get_bucket_encryption({'Bucket': 'elasticbeanstalk-af-south-1-988217567069'}): An error occurred (IllegalLocationConstraintException) when calling the GetBucketEncryption operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

[w0rmr1d3r]

Hello @dhirensham , could this be somehow related to https://github.com/duo-labs/cloudmapper/issues/845 ?

[wonderg]

Same problem on osx

[w0rmr1d3r]

Right @wonderg ! Because the problem is with Python and the code, not with the OS!

[kisst]

The error at leat in my case was here https://github.com/duo-labs/cloudmapper/blob/main/shared/nodes.py#L336 as for GatewayLoadBalancer typed VpcEndpoints you don't get .Groups[].GroupI. I did a dirty try, and bare except with a return of an empty {} only so don't dare to PR it, but wanted to share at least the why...