features Terraform - Downloading GPG Key step gets stuck

Hello,

Since a few days I notice that the GPG Key download step in the Terraform feature hangs:

=> [internal] load metadata for mcr.microsoft.com/devcontainers/base:j  608.5s
 => => extracting sha256:9a586f3d84de83b25cb9ca6d0e733d37d5283da35a837917  0.8s
 => => extracting sha256:5ac951de24f647413f348bfa4183b92c9962b167e0a3106d  0.3s
 => => extracting sha256:1aa90a4f5c993fd6d1e5f0885a1794ee59253ee8d35463ea  0.0s
 => => extracting sha256:074fbd7f2200028d96665f87580813cabf3361b85a4217ce  0.0s
 => => extracting sha256:ee36e8b50d1a0a2f6169666f26f3938fe1ea3efabfb6df01  0.0s
 => => extracting sha256:9e8c9f9ef24f45765c7592d55a952cbe2d0d4fefcd797237  5.5s
 => [dev_containers_feature_content_normalize 2/3] COPY --from=dev_contai  4.8s
 => [dev_containers_feature_content_normalize 3/3] RUN chmod -R 0700 /tmp  1.7s
 => [dev_containers_target_stage  2/12] COPY --from=dev_containers_featur  0.3s
 => [dev_containers_target_stage  3/12] RUN echo "_CONTAINER_USER_HOME=$(  0.7s
 => [dev_containers_target_stage  4/12] RUN cd /tmp/build-features/azure  61.0s
 => [dev_containers_target_stage  5/12] RUN cd /tmp/build-features/github  6.1s
 => [dev_containers_target_stage  6/12] RUN cd /tmp/build-features/kubec  51.5s
 => [dev_containers_target_stage  7/12] RUN cd /tmp/build-features/power  34.3s
 => [dev_containers_target_stage  8/12] RUN cd /tmp/build-features/terr  431.1s
 => => # Downloading terraform...
 => => # TERRAFORM_GPG_KEY=72D7468F
 => => # GPG_KEY_SERVERS=keyserver hkp://keyserver.ubuntu.com:80
 => => # keyserver hkps://keys.openpgp.org
 => => # keyserver hkp://keyserver.pgp.com
 => => # (*) Downloading GPG key...

Screenshot:

I have tried this on various systems/connections and it seems to consistently hang. As you can see, other features install fine.

Any idea how to fix this? Thanks!

Dec 06 '22 12:12 DevSecNinja

I have the same issue. Close remote container on VSCode and choose to rebuild container seem to work, but not very convenient

Dec 06 '22 13:12 SoloW1ngPixy

I am having the same issue. Closing the remote container in VSCode and rebuilding the container did not solve my issue

Dec 06 '22 18:12 sarcasmboy

Same issue with the Python feature getting stuck downloading PGP key

Dec 06 '22 23:12 betancourtca

Here's some additional details. In addition, I have run in a vanilla ubuntu devcontainer without the terraform feature installed and it hangs when executing gpg --verbose --recv-keys 72D7468F which is what appears to be what is executed in the install.sh script during setup.

I've also tried building the devcontainer on multiple different internet connections and am able to visit keyserver.ubuntu.com, keys.openpgp.org, and keyserver.pgp.com in a browser, so I don't believe that it is an external network or routing issue

I am able to reliably reproduce the issues, so if there are any additional debug steps you'd like me to take, let me know.

System Information

Hardware
Processor	12th Gen Intel(R) Core(TM) i7-12700H   2.30 GHz
Installed RAM	32.0 GB (31.7 GB usable)

OS
System type	64-bit operating system, x64-based processor
Edition		Windows 11 Enterprise
Version		22H2
Installed on	‎12/‎4/‎2022
OS build	22621.819
Experience	Windows Feature Experience Pack 1000.22636.1000.0

Docker
Docker Desktop	4.15.0 (93002) 
Backend		WSL 2

devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/ubuntu
{
	"name": "Ubuntu",
	"image": "mcr.microsoft.com/devcontainers/base:jammy",
	"features": {
		"ghcr.io/devcontainers/features/terraform:1": {}
	}

	// Features to add to the dev container. More info: https://containers.dev/features.
	// "features": {},

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "uname -a",

	// Configure tool-specific properties.
	// "customizations": {},

	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
	// "remoteUser": "root"
}

Dec 07 '22 00:12 sarcasmboy

I couldn't reproduce the issue with the dev config mentioned in https://github.com/devcontainers/features/issues/323#issuecomment-1340187092 😓 I've tried multiple times on a Windows and MacOS machine, every time the devcontainer is created/opened fine.

@sarcasmboy Could you provide me information on your Dev Container Extension version & VS Code version?

It is strange that in the original post, the container was able to download gpg keys for the github-cli Feature but couldn't for the terraform Feature 🤔

@joshspicer / @edgonmsft any thoughts on how to fix this? or could you reproduce?

Dec 08 '22 23:12 samruddhikhandale

vscode details

Version: 1.73.1 (user setup)
Commit: 6261075646f055b99068d3688932416f2346dd3b
Date: 2022-11-09T04:27:29.066Z
Electron: 19.0.17
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: No

code --list-extensions --show-versions

[email protected]
[email protected]
[email protected]
[email protected]

Dec 09 '22 01:12 sarcasmboy

I have the same exact configuration. Still can't reproduce.

@sarcasmboy by any change do you have gpg installed on your machine?

I wonder if this could be an issue? 🤔 https://wiki.archlinux.org/title/GnuPG#gpg_hanged_for_all_keyservers_(when_trying_to_receive_keys)

Dec 09 '22 22:12 samruddhikhandale

I have same issue. Below is devcontainer.json

and it stucks only at Terraform Stage.

{
	"build": {
		"dockerfile": "./Dockerfile",
		"context": "."
	},
	"features": {
		"ghcr.io/devcontainers/features/common-utils:1": {
			"installZsh": "true",
			"username": "vscode",
			"uid": "1000",
			"gid": "1000",
			"upgradePackages": "true"
		},
		"ghcr.io/devcontainers/features/git:1": {
			"version": "latest",
			"ppa": "false"
		},
		"ghcr.io/devcontainers/features/azure-cli:1": {
			"version": "latest"
		},
		"ghcr.io/devcontainers/features/terraform:1": {
			"version": "1.1.0",
			"tflint": "0.43.0",
			"tfgrunt": "0.42.3"
		}
	},
	"remoteUser": "vscode"
}

Dec 10 '22 14:12 vnextcoder

@samruddhikhandale I copied the install.sh script into my repository and ran it within a vanilla mcr.microsoft.com/devcontainers/base:jammy devcontainer. When I kill the dirmngr process during the hang, I see the script render an error and then retry until it gives up

output

TERRAFORM_VERSION=1.3.6
TFLINT_VERSION=0.43.0
TERRAGRUNT_VERSION=0.42.3
Downloading terraform...
TERRAFORM_GPG_KEY=72D7468F
GPG_KEY_SERVERS=keyserver hkp://keyserver.ubuntu.com:80
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com
(*) Downloading GPG key...
gpg: keyserver receive failed: End of file
(*) Failed getting key, retring in 10s...
(*) Downloading GPG key...
gpg: keyserver receive failed: End of file
(*) Failed getting key, retring in 10s...
(*) Downloading GPG key...
gpg: keyserver receive failed: End of file
(*) Failed getting key, retring in 10s...
(*) Downloading GPG key...
gpg: keyserver receive failed: End of file
(*) Failed getting key, retring in 10s...
(*) Downloading GPG key...
gpg: keyserver receive failed: End of file
(*) Failed getting key, retring in 10s...
(!) Failed to get gpg key.

Dec 10 '22 16:12 sarcasmboy

I've been trying to think here but without a repro not sure what the issue could be, does it maybe repro localy with Devcontainers extension?

Dec 12 '22 18:12 edgonmsft

@edgonmsft I'm not sure that I understand what you mean by "repro locally with Devcontainers extension". Could you elaborate?

Dec 12 '22 19:12 sarcasmboy

There are so many people able to reproduce this issue. Even running install.sh on plain vanilla container in interactive mode renders same error

Dec 12 '22 23:12 vnextcoder

The feature script succeeds for me. Using the key id gpg --verbose --recv-keys 72D7468F fails. Using the fingerprint from https://www.hashicorp.com/security gpg --verbose --recv-keys "C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F" on the other hand succeeds.

Maybe something with the lookup on the key servers changed. E.g., https://keys.openpgp.org/search?q=72D7468F complains that "Search by Short Key ID is not supported.". Not sure why the feature script still succeeds.

Dec 13 '22 10:12 chrmarti

I am observing a similar issue trying to install Python. I tried originally via Python install script which has worked for me for 1+ years. Yesterday/today the script started hanging at "Downloading GPG key..." so I migrated to use the feature instead and now I am observing that the Python feature is hanging at "Downloading GPG key..."

Dec 15 '22 16:12 austindimmer

@sarcasmboy Can you append --verbose flag to the install script in here? Wondering if that might provide error details.

( echo "${keys}" | xargs -n 1 gpg --verbose -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"

Dec 15 '22 23:12 samruddhikhandale

I have the same issue for several days now.

Dec 16 '22 09:12 daniel-klepacki-medius

@samruddhikhandale No change in the output

output

TERRAFORM_VERSION=1.3.6
TFLINT_VERSION=0.43.0
TERRAGRUNT_VERSION=0.42.5
Downloading terraform...
TERRAFORM_GPG_KEY=72D7468F
GPG_KEY_SERVERS=keyserver hkp://keyserver.ubuntu.com:80
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com
(*) Downloading GPG key...
^C
gpg: signal 2 caught ... exiting

I also added some additional outputs to help debug

echo "(*) Downloading GPG key..."
echo "${keyring_args}"
echo "${keys}"
( echo "${keys}" | xargs -n 1 gpg --verbose -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"

output

TERRAFORM_VERSION=1.3.6
TFLINT_VERSION=0.43.0
TERRAGRUNT_VERSION=0.42.5
Downloading terraform...
TERRAFORM_GPG_KEY=72D7468F
GPG_KEY_SERVERS=keyserver hkp://keyserver.ubuntu.com:80
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com
(*) Downloading GPG key...
keyring_args: 
keys: 72D7468F
^C  
gpg: signal 2 caught ... exiting

Dec 18 '22 19:12 sarcasmboy

I'm not able to reproduce the issue anymore with my own devcontainer and the sample one mentioned by @sarcasmboy. Closing the issue for now, but feel free to comment/reopen if it doesn't work for you.

Dec 19 '22 14:12 DevSecNinja

I'm still getting the same issue, unfortunately.

But I've played around with it and discovered that the issue for me was port 80 on the ubuntu keyserver.

/tmp/tmp-gnupg/dirmngr.conf:

disable-ipv6
log-file /tmp/tmp-gnupg/dirmngr.log
keyserver hkp://keyserver.ubuntu.com:80
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com

Running the command gpg --verbose --recv-keys 72D7468F gave me the following:

gpg: keybox '/tmp/tmp-gnupg/pubring.kbx' created
gpg: no running Dirmngr - starting '/usr/bin/dirmngr'
gpg: waiting for the dirmngr to come up ... (5s)
gpg: connection to dirmngr established

And it just hangs there.

/tmp/tmp-gnupg/dirmngr.log:

2022-12-20 06:52:03 dirmngr[8928.0] permanently loaded certificates: 125
2022-12-20 06:52:03 dirmngr[8928.0]     runtime cached certificates: 0
2022-12-20 06:52:03 dirmngr[8928.0]            trusted certificates: 125 (124,0,0,1)
2022-12-20 06:52:03 dirmngr[8928.0] failed to open cache dir file '/tmp/tmp-gnupg/crls.d/DIR.txt': No such file or directory
2022-12-20 06:52:03 dirmngr[8928.0] creating directory '/tmp/tmp-gnupg/crls.d'
2022-12-20 06:52:03 dirmngr[8928.0] new cache dir file '/tmp/tmp-gnupg/crls.d/DIR.txt' created
2022-12-20 06:52:03 dirmngr[8928.6] resolve_dns_addr for 'keyserver.ubuntu.com': '162.213.33.9'
2022-12-20 06:52:03 dirmngr[8928.6] resolve_dns_addr for 'keyserver.ubuntu.com': '162.213.33.8'
2022-12-20 06:52:03 dirmngr[8928.6] number of system provided CAs: 124

If I change my dirmngr.conf to the following and run it again, it imports it successfully (i.e. remove the :80 on the ubuntu line):

disable-ipv6
log-file /tmp/tmp-gnupg/dirmngr.log
keyserver hkp://keyserver.ubuntu.com
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com

$ gpg --verbose --recv-keys 72D7468F
gpg: keybox '/tmp/tmp-gnupg/pubring.kbx' created
gpg: no running Dirmngr - starting '/usr/bin/dirmngr'
gpg: waiting for the dirmngr to come up ... (5s)
gpg: connection to dirmngr established
gpg: data source: http://162.213.33.9:11371
gpg: armor header: Comment: Hostname:
gpg: armor header: Version: Hockeypuck 2.1.0-189-g15ebf24
gpg: pub  rsa4096/34365D9472D7468F 2021-04-19  HashiCorp Security (hashicorp.com/security) <[email protected]>
gpg: /tmp/tmp-gnupg/trustdb.gpg: trustdb created
gpg: using pgp trust model
gpg: Note: signature key B0B441097685B676 expired Wed 20 Apr 2022 12:08:32 AM UTC
gpg: key 34365D9472D7468F: public key "HashiCorp Security (hashicorp.com/security) <[email protected]>" imported
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: connection to agent established
gpg: Total number processed: 1
gpg:               imported: 1

But I've tried each of the keyservers independently and found issues with all of them.

hkp://keyserver.pgp.com: seems to time out.

gpg: keyserver receive failed: Connection timed out

2022-12-20 07:05:09 dirmngr[12791.6] resolve_dns_addr for 'keyserver.pgp.com': '35.169.187.251'
2022-12-20 07:05:09 dirmngr[12791.6] resolve_dns_addr for 'keyserver.pgp.com': '18.206.36.226'
2022-12-20 07:05:09 dirmngr[12791.6] number of system provided CAs: 124
2022-12-20 07:05:24 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:24 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:24 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:05:39 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:39 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:39 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:05:54 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:54 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:54 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:06:09 dirmngr[12791.6] can't connect to '35.169.187.251': Connection timed out
2022-12-20 07:06:09 dirmngr[12791.6] error connecting to 'http://35.169.187.251:11371': Connection timed out
2022-12-20 07:06:09 dirmngr[12791.6] command 'KS_GET' failed: Connection timed out

hkps://keys.openpgp.org: fails with the short format (works just fine with C874011F0AB405110D02105534365D9472D7468F).

gpg: data source: https://keys.openpgp.org:443
gpg: keyserver receive failed: No data

2022-12-20 07:12:03 dirmngr[14298.6] error accessing 'https://keys.openpgp.org:443/pks/lookup?op=get&options=mr&search=0x72D7468F': http status 400
2022-12-20 07:12:03 dirmngr[14298.6] command 'KS_GET' failed: No data

I've also tried getting the key via curl from the ubuntu keyserver on port 80, and it worked just fine: $ curl 'http://162.213.33.9:80/pks/lookup?op=get&options=mr&search=0x72D7468F'

I don't know what can be done to fix this, though since it seems to be behaving differently for everyone. I just wanted to share my findings.

Dec 20 '22 09:12 eitamal

I am also continuing to experience the issue. Not sure what information or help I can provide to resolve the issue.

Dec 20 '22 14:12 nabil-becker

Reopening this issue for others, but I can't reproduce it anymore. Not sure why.

Dec 20 '22 14:12 DevSecNinja

If it helps ive tried different base images (the latest ubuntu, as well as ubuntu:jammy-20221101)

Im also experiencing this with the ruby feature, so its not just terraform.

Dec 20 '22 15:12 nabil-becker

I was able to work around the issue.

I created a folder in the .devcontainers folder, called it 0terraform.

I then copied the devcontainer-feature.json and install.sh files for the terraform feature into that folder and updated the devcontainer.json to reference it.

It looks like at line 66 of the install.sh, it calls the get_common_setting function which looks like it downloads an env file (https://aka.ms/vscode-dev-containers/script-library/settings.env) and applies it.

I added lines to override the GPG_KEY_SERVERS variable, removing the :80 for the ubuntu keyserver, which resolved the issue.

Is there anyway we can update the https://aka.ms/vscode-dev-containers/script-library/settings.env and remove the :80 reference?

Dec 20 '22 22:12 nabil-becker

@nabil-becker @eitamal Thanks for digging into this!

It's very strange that GPG key gets downloaded if you don't specify a port for Ubuntu Key server. The Ubuntu Key server is always run on port 80 to reduce issues with firewall. I can't find an article which says otherwise.

@nabil-becker / @eitamal Is there a way for you to check if port 80 is blocked on your machine or if VPN is interfering with it?

Dec 20 '22 23:12 samruddhikhandale

Is there anyway we can update the https://aka.ms/vscode-dev-containers/script-library/settings.env and remove the :80 reference?

I could help remove the port 80 suffix from GPG_KEY_SERVERS for the Features, but I'd like to confirm if it solves the issue for everyone and doesn't cause a regression.

Can I get a 👍 if the following dev config works for you?


{
    "image": "mcr.microsoft.com/devcontainers/base:jammy",
    "features": {
        "ghcr.io/samruddhikhandale/test-terraform/terraform:1": {}
    }
}

I have created a temporary terraform Feature on my personal repo which removes :80

https://github.com/samruddhikhandale/test-terraform/blob/main/src/terraform/install.sh#L29

Dec 20 '22 23:12 samruddhikhandale

Created a PR - https://github.com/devcontainers/features/pull/353 Once this one gets merged, I'll update other Features as well.

Dec 20 '22 23:12 samruddhikhandale

Your change seems to be working, thank you

Dec 21 '22 14:12 nabil-becker

Created a PR - #353 Once this one gets merged, I'll update other Features as well.

Merged the PR, closing as resolved. ✨

Feel free to reopen if the issue resurfaces, thanks!

Dec 21 '22 17:12 samruddhikhandale

Thank you. Im also getting the same issue with the ruby and python features. Do we need new issues opened for that?

Dec 21 '22 18:12 nabil-becker

Do we need new issues opened for that?

You may, but I have already started working on updating all other Features. I would report here once completed. Shouldn't take more than an hour.

Dec 21 '22 18:12 samruddhikhandale

features features copied to clipboard

Terraform - Downloading GPG Key step gets stuck

features
features copied to clipboard