devpod
devpod copied to clipboard
loft-sh
Cannot forward gpg-agent with devpod in a "docker-in-docker" container
What happened?
Cannot forward gpg-agent : gpg agent setup is failing
01:47:55 info Run 'ssh admin-devspace-planets-rosterra.devpod' to ssh into the devcontainer
01:47:55 info Starting vscode in browser mode at http://localhost:10800/?folder=/workspace
01:47:55 info Setting up backhaul SSH connection
01:48:00 info gpg: directory '/home/developer/.gnupg' created
01:48:00 info gpg: keybox '/home/developer/.gnupg/pubring.kbx' created
01:48:00 info gpg: /home/developer/.gnupg/trustdb.gpg: trustdb created
01:48:00 info Reverse forwarding local unix//root/.gnupg/S.gpg-agent.extra to remote unix//root/.gnupg/S.gpg-agent.extra
01:48:00 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:01 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:01 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:02 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:02 info Fetch public key: get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:02 info get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
01:48:02 info Error tunneling to container: wait: remote command exited without exit status or exit signal
01:48:02 info tunnel to container: run in container: run gpg agent setup command: Process exited with status 1
01:48:02 error Failed to setup backhaul SSH connection: exit status 1
01:48:03 error Try using the --debug flag to see a more verbose output
01:48:03 fatal outer tunnel: exit status 1
/Projects/@admin/_devspace/_planets/_rosterra # gpg --list-keys
[keyboxd]
---------
pub rsa4096 2025-01-15 [SC]
4A86D720417928707DCA428CA4E2D4B0AE43916C
uid [ultimate] Rishikesavan Ramesh (devspace-key) <[email protected]>
sub rsa4096 2025-01-15 [E]
/Projects/@admin/_devspace/_planets/_rosterra # ps aux | grep gpg
3775 root 0:00 grep gpg
17868 root 0:03 gpg-agent --verbose --daemon --log-file /tmp/gpg-agent.log --allow-preset-passphrase --default-cache-ttl=31536000
/Projects/@admin/_devspace/_planets/_rosterra #
What did you expect to happen instead?
I expect the gpg forwarding works correctly, so i can make signed commits withing the devpods running in a dind container
How can we reproduce the bug? (as minimally and precisely as possible)
My Dockerfile, and startup script of main dind container.
/Projects/@admin/_devspace # ls
Dockerfile README.md _planets daemon.json start_devspace.sh
/Projects/@admin/_devspace # cat Dockerfile
FROM docker:dind
WORKDIR /Projects
RUN apk update; apk add curl npm tmux gpg gpg-agent nano bash
RUN curl -L -o devpod "https://github.com/loft-sh/devpod/releases/latest/download/devpod-linux-amd64" && install -c -m 0755 devpod /usr/local/bin && rm -f devpod
RUN mkdir /etc/docker
COPY daemon.json /etc/docker/daemon.json
RUN devpod provider add docker
RUN devpod context set-options -o EXIT_AFTER_TIMEOUT=false
RUN devpod context set-options default -o SSH_INJECT_GIT_CREDENTIALS=true
RUN devpod context set-options default -o SSH_INJECT_DOCKER_CREDENTIALS=true
RUN devpod context set-options default -o GPG_AGENT_FORWARDING=true
RUN npm install -g @devcontainers/cli
/Projects/@admin/_devspace # cat start_devspace.sh
#!/run/current-system/profile/bin/env bash
docker run -d --name=devspace \
--privileged \
--restart unless-stopped \
--network=host --cap-add=ALL --security-opt seccomp=unconfined \
--volume /home/rishikesavan/Projects:/Projects \
--volume /dev:/dev \
--volume dind-var-lib-docker:/var/lib/docker \
devspace
/Projects/@admin/_devspace # docker -v
Docker version 27.4.1, build b9d17ea
/Projects/@admin/_devspace #
My devcontainer.json:
With or without dind feature in this devcontainer, i could not forward gpg agent.
/Projects/@admin/_devspace # cat _planets/_rosterra/.devcontainer/ros2/devcontainer.json
{
"dockerComposeFile": ["compose.yml"],
"service": "ros-dev",
"runServices": ["ros-dev"],
"shutdownAction": "stopCompose",
"workspaceFolder": "/workspace",
"features": {
// "ghcr.io/devcontainers/features/docker-in-docker:2": {}
},
"customizations": {
"vscode": {
"extensions": [
"ms-vscode.cpptools",
"ms-vscode.cpptools-themes",
"twxs.cmake",
"donjayamanne.python-extension-pack",
"eamodio.gitlens",
"ms-iot.vscode-ros",
"ms-azuretools.vscode-docker",
"ms-vscode.cmake-tools",
"eclipse-cdt.vscode-trace-extension",
"eclipse-cdt.vscode-trace-server",
"redhat.vscode-xml",
"ms-toolsai.jupyter"
]
}
},
"postCreateCommand": "bash /post-create-script.sh"
}
Local Environment:
- DevPod Version: v0.6.8
- Operating System: linux
- ARCH of the OS: AMD64
- Main linux host docker version : Docker version 27.4.1, build b9d17ea
Anything else we need to know?
Hey @RishikesavanRamesh thanks for reporting your issue! Devpod uses a server under the hood running on port 12049 to forward the GPG keys from your local machine to the workspace.
It looks to me like your local machine has some firewall or network policy is blocking port 12049. Can you see if this is the case and if you can remove this? Let me know if that helps!
I get the same error it seems like:
devpod ssh $WORKSPACE_ID --debug
Produces:
...
10:12:06 info Initializing gpg-agent forwarding
10:12:06 info Fetching public key
10:12:06 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
10:12:06 debug Credentials server started on port 12049...
10:12:07 debug Incoming client connection at /gpg-public-keys
10:12:07 debug Successfully wrote back 3256 bytes
10:12:07 info Decoding public key
10:12:07 info Decoding input owner trust
10:12:07 info Stopping container gpg-agent
10:12:07 info Importing gpg public key in container
10:12:07 info Importing gpg owner trust in container
10:12:07 info Ensuring paths existence and permissions
10:12:07 info Ensure paths existence and permissions: exit status 1
10:12:07 debug Connection to container closed
10:12:07 debug done inject
10:12:07 debug done injecting
10:12:07 debug Done InjectAgentAndExecute
10:12:07 error error forwarding /Users/$USER/.gnupg/S.gpg-agent.extra: EOF
10:12:07 error Error tunneling to container: wait: remote command exited without exit status or exit signal
10:12:07 debug Container tunnel exited
10:12:07 debug Tunnel to host closed
10:12:07 debug done exec
10:12:07 fatal run in container: run gpg agent setup command: Process exited with status 1
tunnel to container
github.com/loft-sh/devpod/pkg/tunnel.(*ContainerHandler).Run
/Users/runner/work/devpod/devpod/pkg/tunnel/container.go:131
github.com/loft-sh/devpod/cmd.(*SSHCmd).jumpContainer
/Users/runner/work/devpod/devpod/cmd/ssh.go:276
github.com/loft-sh/devpod/cmd.(*SSHCmd).Run
/Users/runner/work/devpod/devpod/cmd/ssh.go:150
github.com/loft-sh/devpod/cmd.NewSSHCmd.func1
/Users/runner/work/devpod/devpod/cmd/ssh.go:95
github.com/spf13/cobra.(*Command).execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1117
github.com/spf13/cobra.(*Command).Execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1041
github.com/loft-sh/devpod/cmd.Execute
/Users/runner/work/devpod/devpod/cmd/root.go:90
main.main
/Users/runner/work/devpod/devpod/main.go:6
runtime.main
/Users/runner/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:271
runtime.goexit
/Users/runner/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222
Sorry for the late reply @bkneis . I can say that no firewall rules are blocking the port 12049 in my local host, as we can clearly see that gpg forwarding is working in no ide mode, and only causing gpg problem when using code-server.
devspace:/Projects/rosterra# devpod context set-options default -o GPG_AGENT_FORWARDING=true
devspace:/Projects/rosterra# devpod up . --devcontainer-path=.devcontainer/ros2-code-server/devcontainer.json --ide=none
07:56:43 info Workspace rosterra already exists
07:56:43 info Creating devcontainer...
07:56:45 info Setup container...
07:56:46 error unset system credential helper exit status 5
07:56:46 info Run 'ssh rosterra.devpod' to ssh into the devcontainer
devspace:/Projects/rosterra# ssh rosterra.devpod
07:56:57 info Reverse forwarding local unix//root/.gnupg/S.gpg-agent.extra to remote unix//root/.gnupg/S.gpg-agent.extra
$ bash
developer@a3f8669878d8:/workspace$ curl localhost:12049/gpg-public-keys
LS0tLS1CRUdJTiBQR1AgUFVCTElDIEtFWSBCTE9DSy0tLS0tCgptUUlOQkdlc1BvVUJFQUMzaEVESWtJUU1reTREOFRMSFNWWkJKR2Z0OTExcTlMNnJRd0E4QW9VN05Qdy9kS2ZZCldCcGdqT1RHRXRpb2FFRFdFMGZEbWhjN1J6Mk1BaVoyUmdkM21ktMTnpWRnpWcnJidDlLS1BBdVdHVVVhaTY4WVpMRzEzaUJ4d3RTLzVrCmJKMWJrbXRyUjhBdnlTeWswSitpTDdVa1JFYmVNN***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************pWY3NlbW55cGJZVEQ5RWRhMHpkVDkKa200MUhWZlJ2SzczZWkvV0xGdkJRNWhJTzFaL2g2bWc3MXJpK0hxN2ZSU3NJSFB0WUxTd2hLL3d4ZWFoSTkwUgpWQy9LYWdma21jT0EwRWIyVER1MEU2UmFsdVpxOFpUS25ZdTZ4OW92dnpnU3crRmhpa2Q3NUxWL0hlOVlHNEdSCit1ald6S3ZiTzNnUXlpeS81VnBhQnlPemFYSmRSdUI5YlhtQlNGSnJpbkRQOTRiS1phTnk5S1dKWHNzZG96SEkKdEFWNXVmUVZVR3N5SXJEWTlzQTM3dDV5SUdXZm1iMXRQYU9vcTcxNzV3PT0KPTZ0TmcKLS0tLS1F
developer@a3f8669878d8:/workspace$ ^C
developer@a3f8669878d8:/workspace$
exit
$
devspace:/Projects/rosterra# curl localhost:12049
curl: (7) Failed to connect to localhost port 12049 after 0 ms: Could not connect to server
devspace:/Projects/rosterra# curl localhost:12049/gpg-public-keys
curl: (7) Failed to connect to localhost port 12049 after 0 ms: Could not connect to server
devspace:/Projects/rosterra# devpod up . --devcontainer-path=.devcontainer/ros2-code-server/devcontainer.json --ide=openvscode
07:59:19 info Workspace rosterra already exists
07:59:19 info Creating devcontainer...
07:59:20 info Setup container...
07:59:22 error unset system credential helper exit status 5
07:59:22 info Run 'ssh rosterra.devpod' to ssh into the devcontainer
07:59:22 info Starting vscode in browser mode at http://localhost:10800/?folder=/workspace
07:59:22 info Setting up backhaul SSH connection
07:59:24 info gpg: checking the trustdb
07:59:24 info gpg: marginals needed: 3 completes needed: 1 trust model: pgp
07:59:24 info gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
07:59:24 info Reverse forwarding local unix//root/.gnupg/S.gpg-agent.extra to remote unix//root/.gnupg/S.gpg-agent.extra
07:59:24 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
07:59:24 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused```
07:59:24 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
07:59:25 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
07:59:25 info Fetch public key: get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
07:59:25 info get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
07:59:25 info Error tunneling to container: wait: remote command exited without exit status or exit signal
07:59:26 info tunnel to container: run in container: run gpg agent setup command: Process exited with status 1
07:59:26 error Failed to setup backhaul SSH connection: exit status 1
07:59:29 error Try using the --debug flag to see a more verbose output
07:59:29 fatal outer tunnel: exit status 1
devspace:/Projects/rosterra#
I am facing these issues without a docker-in-docker situation, using devpod directly from the command line with no Dockerfile
I am getting the same errors as @RishikesavanRamesh here (similar log with the Failed to setup backhaul SSH connection: exit status 1 and Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys" errors).
It happens when running devpod up . --ide openvscode --recreate. I am facing no errors when running the same command with --ide none.
Also, I don't have any issues SSH-ing into the container as opposed to what @ReneDyhr is reporting. That could be a different issue as neither errors Failed to setup backhaul SSH connection: exit status 1 nor Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys" appear in your case.
Additional context:
- The
devpod upcommand is failing independently of whether a dotfiles repo is attached or not. - I'm using a yubikey with gpg-agent and the pinentry prompt works as expected when running the
devpod upcommand - my repo works with the vscode devcontainer plugin, and it also works with
devpod up . --ide vscode
I'm investigating and will report here my findings. Keen on other people's input too.
Edit: @RishikesavanRamesh do you use a credsStore in your ~/.docker/config.json? (e.g. "credsStore": "pass")? I do (had to install docker-credential-pass from docker-credential-helpers to make it work. It does work except when using --ide openvscode). And do you have GPG_AGENT_FORWARDING set to true? Check with the output of devpod context options.
Here are my settings.
➜ sandbox cat ~/.docker/config.json 12:50:02
{
"auths": {
"https://index.docker.io/v1/": {},
"https://index.docker.io/v1/access-token": {},
"https://index.docker.io/v1/refresh-token": {}
},
"credsStore": "pass"
}⏎
➜ sandbox devpod context options 14:57:59
NAME | DESCRIPTION | DEFAULT | VALUE
--------------------------------+--------------------------------+---------+--------
AGENT_INJECT_TIMEOUT | Specifies the timeout to | 20 | 60
| inject the agent | |
AGENT_URL | Specifies the agent url to use | |
| for DevPod | |
DOTFILES_SCRIPT | Specifies the script to run | |
| after cloning dotfiles repo to | |
| install them | |
DOTFILES_URL | Specifies the dotfiles repo | |
| url to use for DevPod | |
EXIT_AFTER_TIMEOUT | Specifies if DevPod should | true |
| exit the process after the | |
| browser has been idle for a | |
| minute | |
GIT_SSH_SIGNATURE_FORWARDING | Specifies if DevPod should | true |
| automatically detect ssh | |
| signature git setting and | |
| inject ssh signature helper | |
GPG_AGENT_FORWARDING | Specifies if DevPod should | false | true
| do gpg-agent forwarding by | |
| default for ssh | |
REGISTRY_CACHE | Specifies the registry to | |
| use as a build cache, e.g. | |
| gcr.io/my-project/my-dev-env | |
SSH_ADD_PRIVATE_KEYS | Specifies if DevPod should | true | false
| automatically add ssh-keys to | |
| the ssh-agent | |
SSH_AGENT_FORWARDING | Specifies if DevPod should do | true |
| agent forwarding by default | |
| into the workspace | |
SSH_CONFIG_PATH | Specifies the path where the | |
| ssh config should be written | |
| to | |
SSH_INJECT_DOCKER_CREDENTIALS | Specifies if DevPod should | true |
| inject docker credentials into | |
| the workspace | |
SSH_INJECT_GIT_CREDENTIALS | Specifies if DevPod should | true |
| inject git credentials into | |
| the workspace | |
SSH_STRICT_HOST_KEY_CHECKING | Enables strict ssh host key | false |
| checking for all operations | |
TELEMETRY | Specifies if DevPod should | true |
| send telemetry information | |
This issue is stale because it has been open for 60 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}