[Kamal 2] Error: Permission denied (publickey) only for interactive commands

[augustosamame]

I managed (after a few hiccups) to successfully deploy my Rails app to an Ubuntu 24.04 server. The user Im using is ubuntu, and it works fine after I added it to the docker group. I can deploy and I can even run kamal commands remotely like:

kamal app exec 'ruby -v'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [d39a7214] Running docker run --rm --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env JOB_CONCURRENCY="2" --env DB_HOST="mydbhost" --env-file .kamal/apps/scraper_admin/env/roles/web.env --volume scraper_admin_storage:/rails/storage augustosamame/scraper_admin:latest ruby -v on 18.211.91.90
  INFO [d39a7214] Finished in 0.925 seconds with exit status 0 (successful).
App Host: 18.211.91.90
ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux]

however, running any of the interactive commands: kamal console, kamal logs, fails with Permission denied (publickey). The SSH .pem file Im using has to be fine, otherwise it would not work with any commands.

kamal app exec --interactive --reuse "bin/rails console"
Get current version of running container...
  INFO [076e4960] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=scraper_admin --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=augustosamame/scraper_admin:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=scraper_admin --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#scraper_admin-web-}; done on 18.211.91.90
  INFO [076e4960] Finished in 1.591 seconds with exit status 0 (successful).
Launching interactive command with version b7c1aec9139c8084bf9ea67f32fc12283303ad23 via SSH from existing container on 18.211.91.90...
[email protected]: Permission denied (publickey).

These are the relevant parts of my deploy.yml

aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"

# Use a different ssh user than root
ssh:
  keys: [ "/path/to/my/pem" ]
  user: ubuntu
  log_level: debug

In my Dockerfile, I switched from the default rails user to ubuntu:

RUN groupadd --system --gid 1000 ubuntu && \
  useradd ubuntu --uid 1000 --gid 1000 --create-home --shell /bin/bash
RUN chown -R ubuntu:ubuntu db log storage tmp
USER ubuntu

but it did not fix the issue.

[casoetan]

I just hit the same issue a few minutes ago and trying to debug. If I'm able to fix it or identify the issue, I'll ping back

[igrigorik]

Ditto. Looks like something breaks when --interactive is passed in.

[ye-lin-aung]

I am also having same issue

[gioggi]

look here https://github.com/basecamp/kamal/issues/218#issuecomment-2083590982

[taher-tigerlab]

This also seems related to using a custom key in your ssh section of the kamal configuration in my case. I managed to workaround this by using the config option and specifying the key from there instead of specifying it in kamal config

$ cat ~/.ssh/config

IdentityFile /root/.ssh/id_rsa.pem

[PedroAugustoRamalhoDuarte]

For me happens the same error running in kamal 2.2.2, not only for the interactive command.

My hotfix is run ssh-add /path/to/my/pem before running kamal commands.

[dalton-cole]

I added the key to the IdentityFile option for the host in my system ssh config and it solved the issue for now. Will be nice to have the commands use the keys from the deploy.yml.