Discrepancy Between Homebrew vs pip Installations: CKV2 Checks Not Running with Homebrew

[katrinajaneczko]

Description

CKV2 checks do not run when Checkov is installed via Homebrew, but they do run when installed via pip. This discrepancy leads to different outputs between local executions and those in a GitHub Actions (GHA) pipeline, which uses pip for installation.

Steps to Reproduce:

Install Checkov via Homebrew:

brew install checkov

Run Checkov with the following command:

checkov -d . --framework terraform

Observation: CKV2 checks do not appear in the output.

Uninstall Checkov via Homebrew:

brew uninstall checkov

Install Checkov via pip:

pip install checkov

Run the same command again:

checkov -d . --framework terraform

Observation: CKV2 checks now appear in the output, matching the results seen in the GHA pipeline.

Expected Behavior:

Checkov should produce consistent outputs, including CKV2 checks, regardless of whether it is installed via Homebrew or pip.

Actual Behavior:

When installed via Homebrew, Checkov does not run CKV2 checks, leading to discrepancies in the results. Installing via pip resolves this issue and allows CKV2 checks to run as expected.

Environment:

OS: macOS 13.4.1 (M2 chip) Homebrew Version: 4.3.14 Python Version: 3.11.5 pip Version: 24.0 Checkov Version via Homebrew: 3.2.219 Checkov Version via pip: 3.2.219

Additional Context:

I discovered this issue when running Checkov locally and comparing the output to that of a GitHub Actions pipeline. I noticed that when running Checkov checks for a Terraform repository remotely in a GitHub Action workflow, the action was failing due to failed CKV2 checks. However, when running locally via a pre-commit hook, there were no failed checks, and in fact no CKV2 checks were even running at all.

The GHA pipeline installs Checkov using pip, and it correctly runs CKV2 checks, whereas my local installation via Homebrew did not.

I am using this GHA: https://github.com/bridgecrewio/checkov-action Which I found uses a Docker image where Checkov is installed using pip: GitHub Actions Dockerfile.

[99]

I'm experiencing the same issue with the latest versions. I tried a fresh brew install, got 3.2.220, and am seeing the same.

[OfekShimko]

Thank you for reporting this issue. We will work on it next week. In the meantime, we recommend using Pip.

[katrinajaneczko]

Great, thank you!

[rlewis-grax]

Any update on this issue? Still happening in 3.2.330 installed via Homebrew.

EDIT: Issue may be broader than just "CKV2" checks - I'm also seeing the following test failures in GHA but not locally:

• CKV_AWS_18
• CKV_AWS_144
• CKV_AWS_145

[greenflowers]

This seems to be still an issue.

I can reproduce this. In a local checkov installation via brew in version 3.2.360, I do not get any checkov complaints.

In a Bitbucket Pipeline running the upstream image (v.1.97.0), running checkov 3.2.353 installed via pip, I get CKV2 errors.

    pre_commit_check: &pre_commit_check
      step:
        max-time: 20
        size: 2x
        name: "Run Pre-Commit Framework"
        image: ghcr.io/antonbabenko/pre-commit-terraform:v1.97.0
        script:
          - pre-commit run --all-files --show-diff-on-failure

[linbasil]

Reporting that this is still an issue.

With 3.2.400 in both GitHub Actions Ubuntu runners with pip and locally with homebrew, I am seeing CKV* errors with the pip installation, but not with homebrew.

With pip:

"summary": {
    "passed": 34,
    "failed": 5,
    "skipped": 5,
    "parsing_errors": 0,
    "resource_count": 13,
    "checkov_version": "3.2.400"
}

With homebrew:

"summary": {
    "passed": 25,
    "failed": 0,
    "skipped": 5,
    "parsing_errors": 0,
    "resource_count": 13,
    "checkov_version": "3.2.400"
}

[pspittelmeister]

Issue still persists and made me go crazy today :D Is there any chance that this gets fixed?