Allow running detached app commands and follow logs by container ID

[aliismayilov]

This is useful for long running rake tasks or scripts that can be run without having to keep an open connection to servers.

The command that starts a detached container exits immediately and the output leaves the container ID that can be used for following the logs or investigating them later. To support this, I've also extended the kamal logs command with --container-id option.

Example:

❯ kamal app exec --detach 'bin/rails hello'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [21e97fa4] Running docker run --detach --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env-file .kamal/apps/dummy_rails/env/roles/web.env --log-driver "awslogs" --log-opt awslogs-region="eu-central-1" --log-opt awslogs-group="/ec2/dummy-rails" --log-opt awslogs-force-flush-interval-seconds="1" --volume dummy_rails_storage:/rails/storage 233638847975.dkr.ecr.eu-central-1.amazonaws.com/dummy_rails:latest bin/rails hello on 3.77.153.204
  INFO [21e97fa4] Finished in 0.438 seconds with exit status 0 (successful).
App Host: 3.77.153.204
d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090

❯ kamal logs --follow --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t [email protected] -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:20.334253736Z logger says: 3
2024-10-26T13:03:21.334471050Z logger says: 4
2024-10-26T13:03:22.334622132Z logger says: 5
2024-10-26T13:03:23.334916155Z logger says: 6
2024-10-26T13:03:24.335143905Z logger says: 7
2024-10-26T13:03:25.335388754Z logger says: 8
2024-10-26T13:03:26.335653526Z logger says: 9
2024-10-26T13:03:27.335893494Z logger says: 10
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.
❯ kamal logs --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t [email protected] -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.

To support logs inspection, I've also applied kamal logging config to all newly started app exec containers. This resolves https://github.com/basecamp/kamal/issues/1154.

[djmb]

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

[aliismayilov]

Hi @aliismayilov, Sorry for the delay in getting to this! I think we just need some checks for invalid option combinations and we should be good to go!

Thanks for looking into this! I've added an incompatibility check. See if it makes sense.

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

Not anymore! I must've missed them in the avalanche of log lines 😄

[aliismayilov]

I just realized that there will be no output from a detached container in the logs. I created a separate issue: https://github.com/basecamp/kamal/issues/1154

[aliismayilov]

@djmb I've updated the pull request with more changes to support logs inspection for detached containers. In short:

• all newly started app containers get kamal logging configuration applied
• detached containers do not have --rm option. This allows inspection of detached containers even after they finished running
• kamal logs gets a new option --container-id to inspect logs of a specific container

I've also updated the pull request description.

[aliismayilov]

@djmb I've rebased the PR. There shouldn't be any conflicts or failures 😬

[aliismayilov]

OK, I fixed the failing spec and to be sure, the CI passes on my ~~machine~~ account: https://github.com/aliismayilov/kamal/actions/runs/11990309351