checkov fails to scan modules hosted in private repos

[carloscruzadomatrix]

Describe the bug I cant check external repos, the logs print "Failed to download module ". I try this in my pc and with github actions (uses: bridgecrewio/checkov-action@master). The error is the same.

To Reproduce

1. Run cli command checkov -d . --download-external-modules true --download-external-modules .external_modules

2. See error 2021-12-13 12:22:15,601 [ThreadPoolEx] [WARNI] Failed to download module terraform-aws-modules/vpc/aws:3.11.0

[gruebel]

Hi @carloscruzadomatrix can you post the module usage in your Terraform template, then I can better recreate the issue.

[carloscruzadomatrix]

I have 2 modules:

source = "terraform-aws-modules/eks/aws" source = "terraform-aws-modules/vpc/aws"

2021-12-14 16:36:48,043 [MainThread ] [WARNI] Failed to download module terraform-aws-modules/eks/aws:17.24.0 2021-12-14 16:36:48,043 [MainThread ] [WARNI] Failed to download module terraform-aws-modules/vpc/aws:3.11.0

[gruebel]

I tested it and it works fine, my best guess is, that there is some issue with Windows. As I can see you are running the check on your G drive on Windows right? Can you try it on the C drive?

[carloscruzadomatrix]

but the same problem happend in github actions,

[nimrodkor]

Hey @carloscruzadomatrix , did #2118 get this resolved for you?

[tspearconquest]

This is an issue I just ran into

[tspearconquest]

~/test ❯ checkov -d . -o junitxml
2022-04-27 18:43:18,300 [MainThread  ] [WARNI]  Failed to download module gitlab.com/private/project1/azure:0.0.14 (for external modules, the --download-external-modules flag is required)
2022-04-27 18:43:18,301 [MainThread  ] [WARNI]  Failed to download module gitlab.com/private/project2/azure:0.0.3 (for external modules, the --download-external-modules flag is required)
2022-04-27 18:43:18,302 [MainThread  ] [WARNI]  Failed to download module gitlab.com/private/project3/azure:0.0.4 (for external modules, the --download-external-modules flag is required)
2022-04-27 18:43:18,302 [MainThread  ] [WARNI]  Failed to download module gitlab.com/private/project4/azure:0.0.2 (for external modules, the --download-external-modules flag is required)
2022-04-27 18:43:18,302 [MainThread  ] [WARNI]  Failed to download module gitlab.com/private/project5/azure:0.0.20 (for external modules, the --download-external-modules flag is required)
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="0" tests="0" time="0.0">
	<testsuite disabled="0" errors="0" failures="0" name="None scan" skipped="0" tests="0" time="0">
		<properties>
			<property name="directory" value="['.']"/>
			<property name="add_check" value="False"/>
			<property name="list" value="False"/>
			<property name="output" value="['junitxml']"/>
			<property name="output_bc_ids" value="False"/>
			<property name="include_all_checkov_policies" value="True"/>
			<property name="quiet" value="False"/>
			<property name="compact" value="False"/>
			<property name="framework" value="['all']"/>
			<property name="run_all_external_checks" value="False"/>
			<property name="soft_fail" value="False"/>
			<property name="repo_id" value="cli_repo/armed-stack"/>
			<property name="branch" value="master"/>
			<property name="skip_download" value="False"/>
			<property name="no_guide" value="False"/>
			<property name="skip_suppressions" value="False"/>
			<property name="skip_policy_download" value="False"/>
			<property name="skip_fixes" value="False"/>
			<property name="download_external_modules" value="False"/>
			<property name="external_modules_download_path" value=".external_modules"/>
			<property name="evaluate_variables" value="True"/>
			<property name="create_baseline" value="False"/>
		</properties>
	</testsuite>
</testsuites>

Source lines for these modules look as:

source = "gitlab.com/private/project1/azure" source = "gitlab.com/private/project2/azure" source = "gitlab.com/private/project3/azure" source = "gitlab.com/private/project4/azure" source = "gitlab.com/private/project5/azure"

Something I am wondering, if the module is already present in the .terraform directory, can checkov use that instead of downloading from the repo?

[64J0]

I'm having the same problem mentioned by @tspearconquest

[tsmithv11]

This should be resolved thanks to @kartikp10. Let us know if you are still running into issues.

[tspearconquest]

@tsmithv11 do you mind please linking to the pull request which solves this? Thank you in advance.

[kartikp10]

Hey @tspearconquest, the fixes for this came across a few different PRs. Checkov can access private repos using cached git credentials or using a personal access token that's available in an env var. Refer to the Scanning Private Terraform Modules section here. I'm pasting an example of the github-action for the benefit of others on the issue, however, I realize that your use case is not supported at the moment.

name: checkov

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

      - uses: actions/checkout@v3
      - name: Checkov GitHub Action
        uses: bridgecrewio/checkov-action@master
        with:
          directory: .
          download_external_modules: true
          log_level: DEBUG
          github_pat: ${{ secrets.GH_PAT }}
          framework: terraform

In your case, the format gitlab.com/<namespace>/<module-name>/<module-system> refers to a GitLab self-hosted module registry that implements the Terraform Module Registry Protocol, which is not supported in checkov currently (but will be in the future).
We do presently support generic git repo formats (private and public) and almost all other sources (minus buckets and mercurial) mentioned here.

Since this issue refers to private git repo sources, I believe that's fixed and supported. For your use case, the requirement would be to support third-party module registries. Feel free to open a new issue requesting that since I don't think we have one already.

[tspearconquest]

Wow, thank you for the great update! I'll open that issue now!

[faheemgani]

@kartikp10 I'm trying to use this to download modules from my Azure DevOps Private Git Repository using the vcs_base_url, vcs_username and vcs_token environment variables, with no luck. I'm using this with the Checkov Docker Image - like so:

docker run --name checkov_scan_1 --rm --volume 'c:\users\faheem\dev\core\:/core' --env vcs_base_url=https://dev.azure.com/ --env vcs_username=any --env vcs_token=xxx ff86b03e9775 --file '/core/tfplan.json' --repo-root-for-plan-enrichment '/core' --download-external-modules true -o junitxml --soft-fail

I'm also wondering if we can use pre-existing modules that exist on the file system (and inside of the Docker container) rather than downloading them again. In my case all the modules exist inside the Docker container under the /core/.terraform folder so not sure why Checkov tries to download them again.

This is what my module configuration looks like:

module "acr" {
  source = "git::https://dev.azure.com/xxx/Infrastructure/_git/tf-modules//src/modules/acr"
  environment     = var.environment
  project         = var.project
  tenant_id       = var.tenant_id
  subscription_id = var.subscription_id
  rg_name         = var.rg_name
  region          = var.region
  ado_project     = "Infrastructure"
  common_tags     = var.common_tags

  providers = {
    azurerm = azurerm.development
  }

  #checkov:skip=CKV_AZURE_139:Not a premium ACR - this does not support private endpoints
  #checkov:skip=CKV_AZURE_163:Not a premium ACR - this does not support vulnerability scanning
  #checkov:skip=CKV_AZURE_165:Not a premium ACR - this does not support geo-replication
  #checkov:skip=CKV_AZURE_167:Not a premium ACR - this does not support retention periods
}

EDIT: Forgot to post the error that I am getting:

fatal: Authentication failed for 'https://dev.azure.com/xxx/Infrastructure/_git/tf-modules/'
' 2023-04-11 12:18:07,369 [MainThread ] [WARNI] failed to get git::https://dev.azure.com/xxx/Infrastructure/_git/tf-modules because of Cmd('git') failed due to: exit code(128) cmdline: git clone -v --depth=1 -- https://dev.azure.com/xxx/Infrastructure/_git/tf-modules /core/.external_modules/dev.azure.com/xxx/Infrastructure/_git/tf-modules/HEAD stderr: 'Cloning into '/core/.external_modules/dev.azure.com/xxx/Infrastructure/_git/tf-modules/HEAD'... fatal: could not read Username for 'https://dev.azure.com': terminal prompts disabled

[AshishDadhich4h2]

@faheemgani I was trying same (downloading modules from Azure DevOps Private Git Repository) with docker cli with passing --download-external-modules true and got same warnings like [WARNI] Failed to download module

Not sure what is the fix to allow checkov for scanning already downloaded modules in .terraoform directory as part in terrafom init process rather than trying to download and failing without giving any specific reason.

[mikeurbanski1]

@AshishDadhich4h2 you can probably get more info using LOG_LEVEL=DEBUG

[AshishDadhich4h2]

@mikeurbanski1 I don't see LOG_LEVEL option in checkov cli. I am running chekcov docker image and couldn't find any option to pass LOG_LEVEL

[AshishDadhich4h2]

@kartikp10 did you fix your issue ?

[AshishDadhich4h2]

@mikeurbanski1 I don't see LOG_LEVEL option in checkov cli. I am running chekcov docker image and couldn't find any option to pass LOG_LEVEL

Never mind I am able to pass LOG_LEVEL as an env varaiable. Thank you ! But still it doesn't solve the problem