`kamal secrets extract` JSON parse error on shell escaped secrets

[mblayman]

Hi!

I'm trying to use the new secret helpers from Kamal 2 to manage secrets for my project. I tried to follow the style listed in the secrets documentation under "Environment variables", but I'm running into a problem with shell escaping.

I think what is happening is that the raw secrets JSON data is getting to the Ruby parsing code with the shell escape characters still intact. Parsing fails because the JSON parser doesn't accept the backslashes.

I tried to do some puts debugging in the extract method of lib/kamal/cli/secrets.rb to confirm what was going into the JSON.parse call. You can see from my output below that the shell escape characters are still present.

$ SECRETS=$(kamal secrets fetch --adapter 1password --account *** --from some-vault/some-item KAMAL_REGISTRY_PASSWORD)

$ echo $SECRETS
\{\"some-vault/some-item/KAMAL_REGISTRY_PASSWORD\":\"just_alphanumeric_and_underscores\"\}

$ kamal secrets extract KAMAL_REGISTRY_PASSWORD $SECRETS
Hello modified <== my change of `puts "Hello modified"`
\{\"some-vault/some-item/KAMAL_REGISTRY_PASSWORD\":\"just_alphanumeric_and_underscores\"\} <== my change of `puts secrets`
  ERROR (JSON::ParserError): unexpected token at '\{\"some-vault/some-item/KAMAL_REGISTRY_PASSWORD\":\"just_alphanumeric_and_underscores\"\}'

I attempted this on bash and zsh to try to rule out differences in shells.

Is there supposed to be a different way to invoke kamal secrets fetch or kamal secrets extract?

Thanks for the help! I hope this bug report is useful. Please let me know if y'all need any other info.

[mblayman]

I'm not really a Ruby dev, so this might be an inappropriate solution, but Chat GPT helped me devise this alternative implementation of extract:

  desc "extract", "Extract a single secret from the results of a fetch call"
  option :inline, type: :boolean, required: false, hidden: true
  def extract(name, secrets)
    unescaped_secrets = Shellwords.shellsplit(secrets).first
    parsed_secrets = JSON.parse(unescaped_secrets)
    ...

If fetch consistently outputs a shell escaped JSON blob in a single line, I think this might work well.

[mblayman]

Even with my little hack above, something was not working right. I attempted to continue by switching KAMAL_REGISTRY_PASSWORD to use an env variable like KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD. Then I continued with running kamal setup. My setup failed for reasons that I think are app-related, but I when I checked on the web.env file on the server, I found some rather unexpected output. It looks like the command substitution did not happen because I found the following in web.env:

SECRET_KEY=$(kamal secrets extract SECRET_KEY \\{\\"some-vault/some-item/KAMAL_REGISTRY_PASSWORD\\":\\"the_password_was_here\\",\\"some-vault/some-item/SECRET_KEY\\":\\"the_key_was_here\\"\\})

I confirmed in my Django shell that the SECRET_KEY setting was set to that long raw value. My best guess is that not doing the command substitution was also why my KAMAL_REGISTRY_PASSWORD did not work until I changed it to use an env variable.

For completeness, here is my .kamal/secrets (with the sensitive bits removed):

# Secrets defined here are available for reference under registry/password, env/secret, builder/secrets,
# and accessories/*/env/secret in config/deploy.yml. All secrets should be pulled from either
# password manager, ENV, or a file. DO NOT ENTER RAW CREDENTIALS HERE! This file needs to be safe for git.

SECRETS=$(kamal secrets fetch --adapter 1password --account *** --from some-vault/some-item KAMAL_REGISTRY_PASSWORD SECRET_KEY)
#KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD $SECRETS)
KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD
SECRET_KEY=$(kamal secrets extract SECRET_KEY $SECRETS)

[djmb]

Thanks for the report @mblayman!

This worked ok when you specified the kamal secrets commands inside .kamal/secrets, but that was because it "inlines" the calls to the secret commands and the inlining was unescaping the JSON for us. I've moved things around in https://github.com/basecamp/kamal/pull/1008 so it should work ok now both on the command line and in .kamal/secrets

[djmb]

Ok that approach won't work, the problem is that the dotenv gem and the shell don't work in the same way.

https://github.com/basecamp/kamal/pull/1009 adds a kamal secrets print command that can be used for debugging. I'll update the docs to explain this once it is released.

[blvrd]

Ran into the same issue. FWIW, this is my current workaround in .kamal/secrets:

KAMAL_REGISTRY_PASSWORD=$(op read op://vault/item/field)

[mblayman]

@blvrd, yeah, I ended up doing the same thing and used the 1password CLI directly just like you did.

[evdevdev]

@blvrd Thanks for posting your solution. I just burned about 45 knocking my head against this trying to figure out the escape error.

[kmctown]

+1 on the same issue - thanks for the workaround. Only issue is that it is very slow if there are many secrets to load, since we have to do an op call for each secret.

[marciotoshio]

I'm using Bitwarden and have the same workaround, just a little bit more complex

KAMAL_REGISTRY_PASSWORD=$(bw get item ItemName | jq '.fields[] | select(.name=="custom_field_name") | .value' | tr -d '"')

[mikehale]

I just ran into this issue.

[mikehale]

Appending | tr -d '\\' to the end of kamal secrets fetch ... resolved the issue for me, but obviously it would be best if the command was fixed to output valid json.

[adenta]

@mikehale thanks for your solution, it is working for me

[mikehale]

An update... I don't think the commands listed in the kamal secrets file are meant to be run in a console, and in fact they don't work there, however they do work when run in a kamal context. I believe it is using dotenv to parse/execute the file.

[grafst]

An update... I don't think the commands listed in the kamal secrets file are meant to be run in a console, and in fact they don't work there, however they do work when run in a kamal context. I believe it is using dotenv to parse/execute the file.

thank you. I was doing source .kamal/secrets to test. But it actually works with kamal.

[toddkummer]

The solution from @mikehale also worked for me. Looking at the code, it seems like this is what the inline option is for, but I wasn't able to get that to work.

[northeastprince]

@djmb any update on this? seems like the stock example in the docs isn't working

[adenta]

@northeastprince My secrets files look like this if thats helpful

SECRETS=$(kamal secrets fetch --adapter 1password --account EXAMPLE --from Personal/example RAILS_MASTER_KEY KAMAL_REGISTRY_PASSWORD SMTP_EMAIL_ADDRESS SMTP_PASSWORD SMTP_ADDRESS SMTP_PORT SMTP_DOMAIN TWILIO_ACCOUNT_SID OPENROUTER_API_KEY)

GITHUB_TOKEN=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD ${SECRETS})
KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD ${SECRETS})
RAILS_MASTER_KEY=$(kamal secrets extract RAILS_MASTER_KEY ${SECRETS})
SMTP_EMAIL_ADDRESS=$(kamal secrets extract SMTP_EMAIL_ADDRESS ${SECRETS})
SMTP_PASSWORD=$(kamal secrets extract SMTP_PASSWORD ${SECRETS})
SMTP_ADDRESS=$(kamal secrets extract SMTP_ADDRESS ${SECRETS})
SMTP_PORT=$(kamal secrets extract SMTP_PORT ${SECRETS})
SMTP_DOMAIN=$(kamal secrets extract SMTP_DOMAIN ${SECRETS})
TWILIO_ACCOUNT_SID=$(kamal secrets extract TWILIO_ACCOUNT_SID ${SECRETS})
OPENROUTER_API_KEY=$(kamal secrets extract OPENROUTER_API_KEY ${SECRETS})