Issue parsing --env for `app exec`

[mktcode]

I'm trying to reset my production database with this command:

kamal app exec --primary --env=DISABLE_DATABASE_ENVIRONMENT_CHECK:1 'bin/rails db:reset'

It turns into this docker command:

docker run --rm --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env JOB_CONCURRENCY="3" --env WEB_CONCURRENCY="2" --env DB_HOST="xxx.xxx.xxx.xxx" --env-file .kamal/apps/market/env/roles/web.env --env DISABLE_DATABASE_ENVIRONMENT_CHECK="1" --env bin/rails db="reset" --log-opt max-size="10m" --volume market_storage:/rails/storage mktcode/market:latest

Throwing this error:

ERROR (SSHKit::Command::Failed): Exception while executing on host xxx.xxx.xxx.xxx: docker exit status: 125
docker stdout: Nothing written
docker stderr: docker: invalid reference format.

This seems to be wrong: --env bin/rails db="reset"

kamal app exec --help says:

-e, [--env=key:value]     # Set environment variables for the command

Anything I'm doing wrong or is this a bug in the parsing of the env argument?

Update:

When I change the order it works:

kamal app exec --primary 'bin/rails db:reset' --env=DISABLE_DATABASE_ENVIRONMENT_CHECK:1

[mike-weiner]

This does look (and smell) like a parsing issue.

What docker command is generated when kamal app exec --primary 'bin/rails db:reset' --env=DISABLE_DATABASE_ENVIRONMENT_CHECK:1 is used?

After looking at the description of kamal app exec command, I would expect that the command you want to run should be first followed by any options.

weiner ~ % kamal app exec --help 
Usage:
  kamal app exec [CMD...]

...

[mike-weiner]

I tried to recreate this bug with my own deployment by running: kamal app exec --primary --env=TEST:1 '/usr/bin/cat package.json'

The command worked without issue on kamal v2.5.0:

weiner % kamal version                                                    
2.5.0

weiner % kamal app exec --primary --env=TEST:1 '/usr/bin/cat package.json'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [3538b338] Running docker run --rm --network kamal --env PORT="3000" --env TEST="1" --log-opt max-size="10m" registry.hub.docker.com/<REDACTED>/<REDACTED>:latest /usr/bin/cat package.json on <REDACTED>
  INFO [3538b338] Finished in 0.569 seconds with exit status 0 (successful).
...

[mike-weiner]

I was able to recreate this bug. This appears to be caused by a : being in the command to run inside the app container. Here is a basic recreate.

weiner % kamal app exec --primary --env=TEST:1 '/usr/bin/echo :test'      
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [ab8c5ab1] Running docker run --rm --network kamal --env PORT="3000" --env TEST="1" --env /usr/bin/echo ="test" --log-opt max-size="10m" registry.hub.docker.com/<REDACTED>/<REDACTED>:latest  on <REDACTED>
  ERROR (SSHKit::Command::Failed): Exception while executing on host <REDACTED>: docker exit status: 125
docker stdout: Nothing written
docker stderr: docker: invalid reference format.
See 'docker run --help'.

The following variations of the above command succeed:

• kamal app exec --primary '/usr/bin/echo :test' --env=TEST:1
• kamal app exec '/usr/bin/echo :test' --primary --env=TEST:1

[mike-weiner]

I've opened https://github.com/basecamp/kamal/pull/1405 to raise a clearer error message to users.

This appears to be a side effect the env option being of hash type. thor appears to parse everything after --env as key/value pairs.

[aliismayilov]

This seems to be similar to https://github.com/basecamp/kamal/issues/800.

TLDR: The underlying thor gem fails to parse your command. Can you please try?

kamal app exec --primary 'bin/rails db:reset' --env=DISABLE_DATABASE_ENVIRONMENT_CHECK:1

[mike-weiner]

@aliismayilov - Hi, yes thor appears to not handle this specific scenario well. I've already tried a command of that format here: https://github.com/basecamp/kamal/issues/1399#issuecomment-2644400327

It was parsed correctly.

[djmb]

Ugh yeah this is annoying, looks like that's just how the thor hash arguments work.

I guess we could switch them to repeatable string arguments instead and manually parse the key and value so you could do:

kamal app exec --primary --env=TEST:1 --env=TEST2:2 '/usr/bin/echo :test'

That would be a breaking change though so it will need to wait for v3.

[mike-weiner]

I opened https://github.com/basecamp/kamal/pull/1405 awhile back as a stop-gap to try and raise a better error message to the user in this scenario.

Long-term it seems like re-opening https://github.com/rails/thor/issues/881 and getting thor to improve parsing would be best. Not sure if there is some technical challenge as to why thor parses the way it does. If there is some technical challenge, then maybe changing the kamal app exec command in a future release would be another option.