First web container is unhealthy on 5.xx.xxx.xx, not booting any other roles

[deepakmahakale]

I am trying to upgrade kamal to 2.0

Did the necessary changes and ran

kamal upgrade

  INFO [f8a5c978] Running docker exec kamal-proxy kamal-proxy deploy kamal_demo_2-web --target "e19ca135cffb:80" --host "myhostname.com" --tls --deploy-timeout "30s" --drain-timeout "30s" --buffer-requests --buffer-responses --log-request-header "Cache-Control" --log-request-header "Last-Modified" --log-request-header "User-Agent" on 5.xx.xxx.xx
 ERROR Failed to boot web on 5.xx.xxx.xx
  INFO First web container is unhealthy on 5.xx.xxx.xx, not booting any other roles

  INFO [2d5537c0] Running docker container ls --all --filter name=^kamal_demo_2-web-latest$ --quiet | xargs docker logs --timestamps 2>&1 on 5.xx.xxx.xx
  INFO [2d5537c0] Finished in 3.213 seconds with exit status 0 (successful).
 ERROR 2024-09-27T13:22:02.267591832Z => Booting Puma
2024-09-27T13:22:02.267649360Z => Rails 7.1.3 application starting in production
2024-09-27T13:22:02.267653849Z => Run `bin/rails server --help` for more startup options
2024-09-27T13:22:03.064932569Z [1] Puma starting in cluster mode...
2024-09-27T13:22:03.065022850Z [1] * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
2024-09-27T13:22:03.065104904Z [1] *  Min threads: 5
2024-09-27T13:22:03.065200685Z [1] *  Max threads: 5
2024-09-27T13:22:03.065282069Z [1] *  Environment: production
2024-09-27T13:22:03.065386345Z [1] *   Master PID: 1
2024-09-27T13:22:03.065458181Z [1] *      Workers: 2
2024-09-27T13:22:03.065560022Z [1] *     Restarts: (✔) hot (✔) phased
2024-09-27T13:22:03.066126069Z [1] * Listening on http://0.0.0.0:3000
2024-09-27T13:22:03.066317210Z [1] Use Ctrl-C to stop
2024-09-27T13:22:03.079393329Z [1] - Worker 0 (PID: 11) booted in 0.0s, phase: 0
2024-09-27T13:22:03.079564151Z [1] - Worker 1 (PID: 12) booted in 0.0s, phase: 0
  INFO [8246e148] Running docker container ls --all --filter name=^kamal_demo_2-web-latest$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 5.xx.xxx.xx
  INFO [8246e148] Finished in 1.233 seconds with exit status 0 (successful).
 ERROR null
  INFO [958fc459] Running docker container ls --all --filter name=^kamal_demo_2-web-latest$ --quiet | xargs docker stop on 5.xx.xxx.xx
  INFO [958fc459] Finished in 1.734 seconds with exit status 0 (successful).
Releasing the deploy lock...
  ERROR (SSHKit::Command::Failed): Exception while executing on host 5.xx.xxx.xx: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

[kennethgeerts]

I seem to be experiencing the exact same problem with kamal 2. I've tried with the upgrade command, but afterwards also with a 'clean' deploy, where I've removed all docker containers first. I've run in verbose mode and this is the stacktrace printed at the end:

<...>/sshkit-1.23.1/lib/sshkit/command.rb:97:in `exit_status='
<...>/sshkit-1.23.1/lib/sshkit/backends/netssh.rb:185:in `execute_command'
<...>/sshkit-1.23.1/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
<internal:kernel>:90:in `tap'
<...>/sshkit-1.23.1/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
<...>/sshkit-1.23.1/lib/sshkit/backends/abstract.rb:80:in `execute'
<...>/kamal-2.0.0/lib/kamal/cli/app/boot.rb:3:in `execute'
<...>/kamal-2.0.0/lib/kamal/cli/app/boot.rb:57:in `start_new_version'
<...>/kamal-2.0.0/lib/kamal/cli/app/boot.rb:20:in `run'
<...>/kamal-2.0.0/lib/kamal/cli/app.rb:21:in `block (4 levels) in boot'
<...>/kamal-2.0.0/lib/kamal/cli/app.rb:20:in `each'
<...>/kamal-2.0.0/lib/kamal/cli/app.rb:20:in `block (3 levels) in boot'
<...>/sshkit-1.23.1/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
<...>/sshkit-1.23.1/lib/sshkit/backends/abstract.rb:31:in `run'
<...>/kamal-2.0.0/lib/kamal/sshkit_with_ext.rb:118:in `block (2 levels) in execute'

[kobaltz]

I think your error @deepakmahakale could be that you're not routing the traffic to port 3000. By default, kamal-proxy will route the traffic to the container port 80. I wrote about this here https://blog.driftingruby.com/kamal-2-0-issue-with-aws-alb/ but in short, try adding this to your deploy.yml

proxy:
  app_port: 3000

[janpeterka]

Thanks @kobaltz, this helped me greatly. Any idea why 80 is default port?

[kobaltz]

In Rails 8.0.0.beta1 it looks like the Dockerfilr is using thruster which responds on 80.

[igor-alexandrov]

@janpeterka Port 80 is default because Kamal 2 assumes that you have Thruster inside of your container, not just plain Puma. Thruster runs on 80 by default.

[kennethgeerts]

Wouldn't this be useful to mention in the migration section on the kamal site, as I can imagine there will be a lot of folks upgrading while using a Rails < 8 generated Dockerfile?

[igor-alexandrov]

Here is the PR that adds an example of port changing to the generated deploy.yml file. But adding it to the documentation is a also a good idea. I'm on it.

[igor-alexandrov]

@deepakmahakale seems, this can be closed?

[deepakmahakale]

Can't verify right now. Let's close now maybe I will reopen if needed.

[djmb]

Thanks for the report. The docs have now been updated to note that you'll need to specify the app port - see https://kamal-deploy.org/docs/upgrading/configuration-changes/#healthchecks

[defifund]

I think your error @deepakmahakale could be that you're not routing the traffic to port 3000. By default, kamal-proxy will route the traffic to the container port 80. I wrote about this here https://blog.driftingruby.com/kamal-2-0-issue-with-aws-alb/ but in short, try adding this to your deploy.yml我认为你的错误可能是你没有将流量路由到端口 3000。默认情况下， kamal-proxy会将流量路由到容器端口 80。我在这里写了相关内容https://blog.driftingruby.com/kamal -2-0-issue-with-aws-alb/但简而言之，尝试将其添加到您的deploy.yml中

proxy:
  app_port: 3000

Solved my problem, thanks a lot.