[checkov bug] Terraform Checkov not failing properly with --hard-fail-on flag

[CaspervdKerk]

Describe the bug

When running terraform_checkov without any fail flags any matching rule will result in a failed result. However when the --soft-fail and --hard-fail-on MEDIUM flags are enabled, everything passes. Even when violating a HIGH severity rule on purpose.

How can we reproduce it?

Test by violating CKV_AZURE_104 on purpose, which is a HIGH severity rule.

main.tf

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.28.0"
    }
  }
}

resource "azurerm_data_factory" "azure_data_factory" {
  name                   = var.factory_name
  resource_group_name    = var.resource_group_name
  location               = var.location
  public_network_enabled = true

  identity {
    type = "SystemAssigned"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

.pre-commit-config.yaml

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.82.0
    hooks:
      - id: terraform_checkov
        args:
          - --args=--download-external-modules true
          - --args=--quiet
          - --args=--soft-fail
          - --args=--hard-fail-on MEDIUM
          - --args=--check CKV_AZURE_104

Test by using pre-commit run terraform_checkov --all-files. When the flags are enabled the pre-commit hook will pass. When the lines - --args=--soft-fail and - --args=--hard-fail-on MEDIUM are commented out, the hook will fail.

Environment information

• OS: MacOS Ventura 13.5.1

• uname -a and/or systeminfo | Select-String "^OS" output:

Darwin MB0330 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64 x86_64

• Tools availability and versions:

GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin22)
pre-commit 3.4.0
Terraform v1.5.7
python SKIPPED
Python 3.11.5
checkov 2.4.30
terraform-docs version v0.16.0 darwin/amd64
terragrunt SKIPPED
terrascan terrascan SKIPPED
TFLint version 0.48.0
+ ruleset.terraform (0.4.0-bundled)
tfsec tfsec SKIPPED
tfupdate tfupdate SKIPPED
hcledit hcledit SKIPPED

• .pre-commit-config.yaml: (same as above)

.pre-commit-config.yaml

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.82.0
    hooks:
      - id: terraform_checkov
        args:
          - --args=--download-external-modules true
          - --args=--quiet
          - --args=--soft-fail
          - --args=--hard-fail-on MEDIUM
          - --args=--check CKV_AZURE_104

[MaxymVlasov]

You're right. That's a bug in checkov - it returns wrong exit code

[MaxymVlasov]

If you run checkov -d "$(pwd)" --hard-fail-on MEDIUM - you'll get error message and exit code 0

[MaxymVlasov]

@CaspervdKerk please open the issue in https://github.com/bridgecrewio/checkov/ It also break their checkov pre-commit hook, as far as I can see

https://github.com/bridgecrewio/checkov/blob/05cf950fd9f1d1eaba2998d9b845b62494df060c/.pre-commit-hooks.yaml#L6-L15

[CaspervdKerk]

Thank you for confirming my findings! I'll raise this bug on the checkov repo as well. There is definitely some strange behavior going on. During my testing I found my code also triggered CKV_AZURE_103, which is a LOW severity check. Using the pre-commit hook with --soft-fail and --hard-fail-on MEDIUM Checkov passes, as is expected. However, when I use the local installed Checkov CLI it fails on this LOW severity check, even though the --soft-fail and --hard-fail-on MEDIUM flags are passed...

[MaxymVlasov]

From https://github.com/bridgecrewio/checkov/issues/5560#issuecomment-1722313286 :

Severities only work with an API key, that's why it doesn't work as expected.