bridgecrewio
Unable to load module when referencing a submodule
Hello, I'm using a Terraform Code with Modules from another Repo, and I have created a user/token for this download.
Terraform is working fine, but when I tried to use checkov to validate this main.tf which uses a source from another git, I received this error below.
[0/6]2022-06-29 01:18:00,393 [MainThread ] [WARNI] Unable to load module (source="git::https://{user}:{token}@github.{provider}.com/{repo}/terraform-helloworld-modules.git//modules" version="latest"): [Errno 2] No such file or directory: '/tf/.external_modules/github.{provider}.com/{repro}/terraform-helloworld-modules/HEAD/modules'
My problem, in this case, occurred because my customer is using one repo for terraform modules (common modules) in different subdirectories.
In this case, in main.tf we used a "//modules" to choose with the "module" we are using and because of this, the error occurred.
Can someone help me?
hey @alexandrezanetti thanks for reaching out. Can you add an example of the module usage in one of the TF files. If there are sensible data, just replace them with something 😄 then it is easier for me to reproduce it.
Hi Below you can see my main.tf
module "storage_account" {
source = "git::https://{user}:{token}@github.ibm.com/alexandre-zanetti/terraform-helloworld-modules.git//modules"
# name = var.storage_account_name
# resource_group_name = var.resource_group_name
# location = var.location
}
====================================================
And below you can see my module zzzcheckov.tf:
resource "azurerm_storage_account" "storage_account" {
name = var.storage_account_name
resource_group_name = var.resource_group_name
location = var.location
account_replication_type = var.account_replication_type
account_tier = var.account_tier
}
====================================================
I think that the problem on Checkov code is related to "//modules" subdirectory on the code below:
module "storage_account" { source = "git::https://{user}:{token}@github.ibm.com/alexandre-zanetti/terraform-helloworld-modules.git//modules"
But this strategy is accepted on Terraform.
Let me know if you need more details.
just to clarify it is exactly written with the placeholders for user and token or you just removed them for posting it here
module "storage_account" {
source = "git::https://{user}:{token}@github.ibm.com/alexandre-zanetti/terraform-helloworld-modules.git//modules"
...
}
and not something like
module "storage_account" {
source = "git::https://user123:[email protected]/alexandre-zanetti/terraform-helloworld-modules.git//modules"
...
}
because the error you get has nothing to do with accessing the submodule, we just couldn't download the repo at all.
Hi, I have only remove my credentials, because here is public. I'm using like you share: module "storage_account" { source = "git::https://user123:[email protected]/alexandre-zanetti/terraform-helloworld-modules.git//modules" ... }
hmmm 🤔 I created a private repo with a simple module and it all works well. I think something is blocking the download. Can you run the checkov command with DEBUG logging enabled, maybe we get a bit more info, what could be wrong.
LOG_LEVEL=DEBUG checkov -d . --download-external-modules true --framework terraform
Hi Gruebel,
Have you create a subdirectory for example moduloRG, moduleStorageAccount and try to use checkov validations (consequently download submodules)?
"git::https://user123:[email protected]/alexandre-zanetti/terraform-helloworld-modules.git//moduleStorageAccount"
That's the problem.
Let me know if you could reproduce this.
yeah I did 🙂
module "bucket" {
source = "git::https://user123:[email protected]/gruebel/private-tf-modules.git//bucket"
name = "anton-1234567890"
}
and then the result was
_ _
___| |__ ___ ___| | _______ __
/ __| '_ \ / _ \/ __| |/ / _ \ \ / /
| (__| | | | __/ (__| < (_) \ V /
\___|_| |_|\___|\___|_|\_\___/ \_/
By bridgecrew.io | version: 2.1.20
terraform scan results:
Passed checks: 5, Failed checks: 6, Skipped checks: 0
Check: CKV_AWS_41: "Ensure no hard coded AWS access key and secret key exists in provider"
PASSED for resource: aws.default
File: /main.tf:1-4
Guide: https://docs.bridgecrew.io/docs/bc_aws_secrets_5
Check: CKV_AWS_70: "Ensure S3 bucket does not allow an action with any Principal"
PASSED for resource: module.storage_account.aws_s3_bucket.this
File: /.external_modules/github.com/gruebel/private-tf-modules/HEAD/bucket/main.tf:1-3
Calling File: /main.tf:6-10
Guide: https://docs.bridgecrew.io/docs/bc_aws_s3_23
...
Hi Gruebel,
Next Monday I asked to Fernando (who works with me) to reexecute and share the result in this issue.
Sorry about that, because I'm on vacation (since last Friday night) and I'm abroad of my country.
Looking for your command, I forget to use "--framework terraform".
I will ask Fernando to try to execute with this.
the command "--framework terraform" was used Debug level of the code: 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV_AZURE_24: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_8: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_1: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_21: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_20: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_4: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_5: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_2: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_3: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_12: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_AZURE_9: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_GCP_7: True 2022-07-07 11:37:42,517 [MainThread ] [DEBUG] Should run check CKV2_GCP_5: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_9: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_6: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_8: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_4: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_2: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_1: True 2022-07-07 11:37:42,518 [MainThread ] [DEBUG] Should run check CKV2_GCP_3: True 2022-07-07 11:37:42,518 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_28 2022-07-07 11:37:42,519 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_20 2022-07-07 11:37:42,519 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_7 2022-07-07 11:37:42,520 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_4 2022-07-07 11:37:42,520 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_29 2022-07-07 11:37:42,520 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_35 2022-07-07 11:37:42,522 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_15 2022-07-07 11:37:42,521 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_33 2022-07-07 11:37:42,522 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_16 2022-07-07 11:37:42,521 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_34 2022-07-07 11:37:42,523 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_32 2022-07-07 11:37:42,523 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_10 2022-07-07 11:37:42,523 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_37 2022-07-07 11:37:42,524 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_9 2022-07-07 11:37:42,524 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_18 2022-07-07 11:37:42,524 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_19 2022-07-07 11:37:42,525 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_2 2022-07-07 11:37:42,525 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_3 2022-07-07 11:37:42,525 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_36 2022-07-07 11:37:42,526 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_14 2022-07-07 11:37:42,530 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_145 2022-07-07 11:37:42,526 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_22 2022-07-07 11:37:42,526 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_30 2022-07-07 11:37:42,528 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_27 2022-07-07 11:37:42,528 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_8 2022-07-07 11:37:42,528 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_23 2022-07-07 11:37:42,528 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_19 2022-07-07 11:37:42,529 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_6 2022-07-07 11:37:42,529 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_18 2022-07-07 11:37:42,529 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_144 2022-07-07 11:37:42,529 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_21 2022-07-07 11:37:42,526 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_21 2022-07-07 11:37:42,530 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_20 2022-07-07 11:37:42,530 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AWS_57 2022-07-07 11:37:42,531 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_5 2022-07-07 11:37:42,532 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_1 2022-07-07 11:37:42,536 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_19 2022-07-07 11:37:42,532 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_11 2022-07-07 11:37:42,533 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_31 2022-07-07 11:37:42,533 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_6 2022-07-07 11:37:42,533 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AZURE_120 2022-07-07 11:37:42,534 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_7 2022-07-07 11:37:42,534 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_10 2022-07-07 11:37:42,534 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_15 2022-07-07 11:37:42,535 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_13 2022-07-07 11:37:42,535 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AZURE_119 2022-07-07 11:37:42,535 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_18 2022-07-07 11:37:42,532 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AWS_12 2022-07-07 11:37:42,536 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_14 2022-07-07 11:37:42,536 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_22 2022-07-07 11:37:42,536 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_11 2022-07-07 11:37:42,537 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_16 2022-07-07 11:37:42,537 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_17 2022-07-07 11:37:42,540 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AZURE_23 2022-07-07 11:37:42,540 [ThreadPoolEx] [DEBUG] Running graph check: CKV_AZURE_24 2022-07-07 11:37:42,541 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_8 2022-07-07 11:37:42,541 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_1 2022-07-07 11:37:42,541 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_21 2022-07-07 11:37:42,542 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_20 2022-07-07 11:37:42,542 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_4 2022-07-07 11:37:42,542 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_5 2022-07-07 11:37:42,542 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_2 2022-07-07 11:37:42,543 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_3 2022-07-07 11:37:42,543 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_12 2022-07-07 11:37:42,543 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_AZURE_9 2022-07-07 11:37:42,543 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_7 2022-07-07 11:37:42,544 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_5 2022-07-07 11:37:42,544 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_9 2022-07-07 11:37:42,544 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_6 2022-07-07 11:37:42,544 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_8 2022-07-07 11:37:42,545 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_4 2022-07-07 11:37:42,545 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_2 2022-07-07 11:37:42,545 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_1 2022-07-07 11:37:42,545 [ThreadPoolEx] [DEBUG] Running graph check: CKV2_GCP_3 2022-07-07 11:37:42,550 [MainThread ] [DEBUG] Getting exit code for report terraform 2022-07-07 11:37:42,550 [MainThread ] [DEBUG] In get_exit_code; soft_fail: False, soft_fail_on: None, hard_fail_on: None, hard_fail_on_parsing_errors: False 2022-07-07 11:37:42,550 [MainThread ] [DEBUG] No failed checks, or soft_fail is True and soft_fail_on and hard_fail_on are empty - returning 0
This happens without DEGUB LOG:
2022-07-07 12:03:35,474 [MainThread ] [ERROR] failed to get git::https://bitbucket.organization.com.br:8443/scm/cpppv/repository because of Cmd('git') failed due to: exit code(128) cmdline: git clone -v --depth=1 https://bitbucket.organization.com.br:8443/scm/cpppv/repository /tf/internalrepository/azure-eventhub/.external_modules/bitbucket.organization.com.br:8443/scm/cpppv/repository/HEAD stderr: 'Cloning into '/tf/internalrepository/azure-eventhub/.external_modules/bitbucket.organization.com.br:8443/scm/cpppv/repository/HEAD'... fatal: could not read Username for 'https://bitbucket.organization.com.br:8443': No such device or address 2022-07-07 12:03:35,763 [MainThread ] [WARNI] Failed to download module git::https://user123:[email protected]:8443/scm/cpppv/repository.git//Azu-RG:None
Code used in main.tf:
module "rg" { source = "git::https://user123:[email protected]:8443/scm/cpppv/repository.git//Azu-RG"
rg_name = "rgsqlpoc" }
Hello ,
Even I am facing the same issue .
Checkov works fine with below repo structure (for terraform common modules )
whereas it fails when the repo is structure is like below .
We are following the 2nd repo structure to maintain all our common terraform modules so can someone help ?
We are following the 2nd repo structure to maintain only main.tf, all our common terraform modules following the first one.
What worked for me is to create a dummy TF module which will download all the modules you have in the remote repo. Since we specify the top level folder in the repo, then all modules are downloaded locally and cached.
# apparently when specifying a module name in the Git URL checkov can no longer parse the token
module "checkov_dummy_download_module" {
source = "git::https://oauth2:xxx@yyy/path/to/git/repo.git//?ref=v1"
}
Once that is done, checkov will no longer follow the git URL to download the sub-modules in the remote repo but use the ones it downloaded locally.
module "my_module" {
source = "git::https://oauth2:xxx@yyy/path/to/git/repo.git//my-module?ref=v1"
}
Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
We are experiencing a similar issue and we think it's due to how we tag our terraform modules. Unfortunately we have a mono-repo which consists of all our modules. These modules make use of external modules. When we enable the flag to download external modules we get the following message back -
2023-02-15 11:43:00,507 [MainThread ] [DEBUG] Cache miss for [email protected]:xxxxxx-Infra/terraform-modules.git//modules//account?ref=account-v1.0.0:latest
2023-02-15 11:43:00,548 [MainThread ] [WARNI] Module [email protected]:xxxxxx-Infra/terraform-modules.git//modules//account?ref=account-v1.0.0:latest failed to load via <class 'checkov.terraform.module_loading.loaders.github_loader.GithubLoader'>
2023-02-15 11:43:00,[54](https://github.com/xxxxxx-Infra/aws-accounts/actions/runs/4183495950/jobs/7247921624#step:8:55)9 [MainThread ] [WARNI] Unable to load module ([email protected]:xxxxxx-Infra/terraform-modules.git//modules//account?ref=account-v1.0.0:None): invalid git url
Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!
Closing issue due to inactivity. If you feel this is in error, please re-open, or reach out to the community via slack: https://slack.bridgecrew.io Thanks!
Hi, i'm having the same issue here with my bitbucket private repos:
2023-11-09 14:22:38,067 [MainThread ] [WARNI] Failed to download module [email protected]:<redacted>/repo-name.git?ref=main:None
2023-11-09 14:22:38,068 [MainThread ] [INFO ] Downloading module [email protected]:<redacted>/repo-name.git?ref=main:None
Experiencing the same issue
2024-03-26 14:50:59,619 [MainThread ] [WARNI] Unable to load module - source:
git::https://gitlab.private.com/common-infra/modules.git//security/iam/oidc-role?ref=v2.1.6, version: latest, error:
[Errno 2] No such file or directory: '/builds/project-infra/project-
terragrunt/modules/.external_modules/gitlab.private.com/common-infra/modules/v2.1.6/security/iam/oidc-role'
module "iam_oidc_role" {
source = "git::https://gitlab.private.com/common-infra/modules.git//security/iam/oidc-role?ref=v2.1.6"
...
...
}
My checkov command:
checkov --repo-root-for-plan-enrichment ${CI_PROJECT_DIR}/modules --download-external-modules true \
-f ${CI_PROJECT_DIR}/${MODULE}/plan.json -o junitxml --soft-fail > \
${CI_PROJECT_DIR}/${MODULE}/checkov.test.xml
Is it possible to reopen this issue or do you know a solution? @gruebel
Thanks 🙏
