checkov
checkov copied to clipboard
Published
20 hours ago •
bridgecrewio
bridgecrewio
Inconsistent output for identical code scan
Describe the issue I have been running checkov locally (installed on MAC) and in the docker container but getting different results each time I can the exact same code.
The output below is for 4 executions of Checkov on the exact same code but with 3 different results. This run was just using custom checks only.
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test1: "Check that all resources are tagged with the key - slz:test1 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test2: "Check that all resources are tagged with the key - slz:test2 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test4: "Check that all resources are tagged with the key - slz:test4 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Passed checks: 4, Failed checks: 4, Skipped checks: 0
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Passed checks: 7, Failed checks: 1, Skipped checks: 0
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Passed checks: 7, Failed checks: 1, Skipped checks: 0
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --config-file /tf/config/checkov/config.yaml --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-check CKV2_AWS*,CKV_AWS*
terraform scan results:
Check: CKV3_AWS_IT_DATA_TAGS_test1: "Check that all resources are tagged with the key - slz:test1 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test2: "Check that all resources are tagged with the key - slz:test2 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Check: CKV3_AWS_IT_DATA_TAGS_test3: "Check that all resources are tagged with the key - slz:test3 and have a valid value"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV3_AWS_IT_DATA_TAGS_test4: "Check that all resources are tagged with the key - slz:test4 and have a valid value"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Passed checks: 11, Failed checks: 5, Skipped checks: 0
If I exclude the custom checks and don't use the config.yaml, so just a normal terraform run with CLI switches then I still get inconsistent results between runs.
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-download --summary-position bottom --download-external-modules true --framework terraform --compact --quiet
terraform scan results:
Check: CKV_AWS_109: "Ensure IAM policies does not allow permissions management / resource exposure without constraints"
FAILED for resource: aws_iam_policy_document.rds_kms_policy
File: /data.tf:54-133
Check: CKV_AWS_111: "Ensure IAM policies does not allow write access without constraints"
FAILED for resource: aws_iam_policy_document.rds_kms_policy
File: /data.tf:54-133
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team3.tf:5-84
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team3.tf:5-84
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team6.tf:5-85
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team6.tf:5-85
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team5.tf:5-84
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /team5.tf:5-84
Check: CKV_AWS_226: "Ensure DB instance gets all minor upgrades automatically"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /openbanking_pf.tf:5-85
Check: CKV_AWS_118: "Ensure that enhanced monitoring is enabled for Amazon RDS instances"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster_instance.cluster_instance
File: /../../modules/rds/main.tf:215-250
Calling File: /openbanking_pf.tf:5-85
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Passed checks: 41, Failed checks: 26, Skipped checks: 0
┌─[USER][MACHINE][±][checkov-testing {2} S:8 U:13 ?:6 ✗][~/dev/test-terraform]
└─▪ docker run --tty --rm --volume /Users/USER/dev/test-terraform/:/tf --workdir /tf/infra/rds bridgecrew/checkov --directory /tf/infra/rds --var-file /tf/infra/rds/common.tfvars.json --var-file /tf/config/dev/dev.tfvars --skip-download --summary-position bottom --download-external-modules true --framework terraform --compact --quiet
terraform scan results:
Check: CKV_AWS_109: "Ensure IAM policies does not allow permissions management / resource exposure without constraints"
FAILED for resource: aws_iam_policy_document.rds_kms_policy
File: /data.tf:54-133
Check: CKV_AWS_111: "Ensure IAM policies does not allow write access without constraints"
FAILED for resource: aws_iam_policy_document.rds_kms_policy
File: /data.tf:54-133
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_8: "Ensure that RDS clusters has backup plan of AWS Backup"
FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team1_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team2_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team3_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team4_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team5_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team6_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team7_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Check: CKV2_AWS_27: "Ensure Postgres RDS as aws_rds_cluster has Query Logging enabled"
FAILED for resource: module.rds_team1_playground_team_encrypted.aws_rds_cluster.cluster
File: /../../modules/rds/main.tf:6-52
Passed checks: 37, Failed checks: 18, Skipped checks: 0
Additional context This happens if I run the commands directly on the MAC and not within the docker container.
Updated to latest version 2.3.150 today and it's better, as in most consecutive runs are the same, 1 out of 5 are different.
hey @nicholas-marchini thanks for reaching out.
It looks like the inconsistency comes from the massive usage of the same module rds not so surprising for me. Additionally using multiple tfvars files doesn't make it easier.
@gruebel Thanks for the reply. We operate a multi-tenent AWS account and do have the need to use multiple tfvars files right now.
Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
hey @nmarchini
Do you have the same setup, multiple times using the same module? If yes, how many times is it?
@gruebel We call the module 9 times. I have used the latest docker image and am still getting inconsistent results
Same issue here - different tests are running on my local machine (Mac) than in a container AWS CodeBuild is managing (more tests are checked in the container). However in my case the issue is happening even within just a single module (not referencing any other modules).
Is there way to define/enforce a list of tests that you want checkov to run? Something like checkov -d . --run-tests: 'CVE_AWS'? How does checkov determine what tests to run when presented a file or directory?
I've given up on checkov as having this issue open since Apr 2, 2023 is poor, since Checkov got bought the level of response and interaction to issues posted here has dropped drastically. We moved to TFsec now and are very happy with it.
Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at codifiedsecurity.slack.com Thanks!