"Host key verification failed" during kamal setup

[stephensamra]

Attempting to run kamal setup against a fresh server fails to create the remote builder:

INFO [048248a6] Running docker context create kamal-server-native-remote-amd64 --description 'kamal-server-native-remote amd64 native host' --docker 'host=ssh://[email protected]' ; docker buildx create --name kamal-server-native-remote kamal-server-native-remote-amd64 --platform linux/amd64 on localhost
ERROR Couldn't create remote builder: Host key verification failed.

Full output

Acquiring the deploy lock...
  INFO [407c59d6] Running docker -v on xxx.xxx.xxx.xxx
  INFO [407c59d6] Finished in 0.091 seconds with exit status 0 (successful).
Log into image registry...
  INFO [8a6395f3] Running docker login -u [REDACTED] -p [REDACTED] on localhost
  INFO [8a6395f3] Finished in 1.433 seconds with exit status 0 (successful).
  INFO [871216aa] Running docker login -u [REDACTED] -p [REDACTED] on xxx.xxx.xxx.xxx
  INFO [871216aa] Finished in 0.881 seconds with exit status 0 (successful).
Build and push app image...
  INFO [35637a22] Running docker --version && docker buildx version on localhost
  INFO [35637a22] Finished in 0.088 seconds with exit status 0 (successful).
The following paths have uncommitted changes:
 ...
  INFO [0317c4ce] Running docker buildx build --push --platform linux/amd64 --builder kamal-server-native-remote -t xxx/xxx -t xxx/xxx:latest --label service="server" --file docker/server/Dockerfile . on localhost
 DEBUG [0317c4ce] Command: docker buildx build --push --platform linux/amd64 --builder kamal-server-native-remote -t xxx/xxx -t xxx/xxx:latest --label service="server" --file docker/server/Dockerfile .
 DEBUG [0317c4ce] 	ERROR: no builder "kamal-server-native-remote" found
 ERROR Missing compatible builder, so creating a new one first
  INFO [048248a6] Running docker context create kamal-server-native-remote-amd64 --description 'kamal-server-native-remote amd64 native host' --docker 'host=ssh://[email protected]' ; docker buildx create --name kamal-server-native-remote kamal-server-native-remote-amd64 --platform linux/amd64 on localhost
 ERROR Couldn't create remote builder: Host key verification failed.
  INFO [1f5e7abd] Running docker image rm --force xxx/xxx on xxx.xxx.xxx.xxx
  INFO [1f5e7abd] Finished in 0.098 seconds with exit status 0 (successful).
  INFO [f6b8afc1] Running docker pull xxx/xxx on xxx.xxx.xxx.xxx
  Finished all in 6.4 seconds
Releasing the deploy lock...
  Finished all in 7.5 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host xxx.xxx.xxx.xxx: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error response from daemon: manifest for xxx/xxx not found: manifest unknown: manifest unknown

I can SSH into the server without issue.

I tried running ssh-add and ssh-agent -s but that did not seem to change anything.

I also tried to run the failing command on its own and it did seem to work:

docker context create kamal-server-native-remote-amd64 \
  --description 'kamal-server-native-remote amd64 native host' \
  --docker 'host=ssh://[email protected]' \
  kamal-server-native-remote-amd64

Successfully created context "kamal-server-native-remote-amd64"

I wasn't having this issue the last time I ran mrsk setup, which was last Friday (2023-08-18).

Any help would be appreciated, thanks in advance.

[n4cr]

Facing the same issue when using a remote builder

[aitoehigie]

I am experiencing the same.

[pjar]

Facing the same issue - I can ssh to the user from remote builder host but kamal seems to be unable to do the same. Running just the failing command works - as for the OP.

[fennekit]

If you are running the docker kamal container, it is trying to find ssh keys in /root/.ssh. And there are no keys there since it is not mapped to anything. Add your .ssh dir as a volume (probably not the safest solution, but it works)

alias kamal='docker run -it --rm -v "${PWD}:/workdir" -v "${HOME}/.ssh:/root/.ssh" -v "/run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock" -e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/basecamp/kamal:latest'

[sbounmy]

thanks @fennekit it worked. but definitely looking for a safer way to handle this on the long term :)

[abratashov]

To solve such an issue, firstly what I do after fresh server setup:

ssh [email protected]
mkdir ~/.ssh
touch ~/.ssh/authorized_keys
exit

then on local machine:

local$ cat ~/.ssh/id_ed25519.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'

[gryphon]

For better safety you can only pass known hosts as a volume: "${HOME}/.ssh/known_hosts:/root/.ssh/known_hosts"