I don't think this is the same as https://github.com/basecamp/kamal/issues/1007 because in general my secrets are working.

The only secret that isn't working is the one that holds a JSON value which contains my Google service account config. Locally, my features involving the service account are working and i'm using the same secrets manager so I think there is something weird going on with how this secret is handled during a kamal deploy.

Context

Kamal 2.5.2
Rails 8.0.1
agent: Doppler

# kamal/secrets

SECRETS=$(kamal secrets fetch --adapter doppler --from project/prd KAMAL_REGISTRY_PASSWORD RAILS_MASTER_KEY GOOGLE_SERVICE_ACCOUNT_CONFIG)
KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD ${SECRETS})
RAILS_MASTER_KEY=$(kamal secrets extract RAILS_MASTER_KEY ${SECRETS})
GOOGLE_SERVICE_ACCOUNT_CONFIG=$(kamal secrets extract GOOGLE_SERVICE_ACCOUNT_CONFIG ${SECRETS})

# EXAMPLE - contents of the service account config

GOOGLE_SERVICE_ACCOUNT_CONFIG={
  "type": "service_account",
  "project_id": "project-dev",
  "private_key_id": "*****",
  "private_key": "-----BEGIN PRIVATE KEY-----\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n*****\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "123451133783890925820",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/connect%40project-dev.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

DIFF

Here's the difference in what I'm seeing in console locally vs on the server for ENV['GOOGLE_SERVICE_ACCOUNT_CONFIG']

	local	server
working	yes	no
`ENV['GO...']`	"{\n \"type\": \"service_account\",\n \"project_id\": \"project-dev\",\n \"private_key_id\": \"***\",\n \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n*\\n*\\n ... \\n-----END PRIVATE KEY-----\\n\",\n \"client_email\": \"[email protected]\",\n \"client_id\": \"****\",\n \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n \"token_uri\": \"https://oauth2.googleapis.com/token\",\n \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/connect%40project-dev.iam.gserviceaccount.com\",\n \"universe_domain\": \"googleapis.com\"\n}"	"{\\n \"type\": \"service_account\",\\n \"project_id\": \"project-dev\",\\n \"private_key_id\": \"*****\",\\n \"private_key\": \"-----BEGIN PRIVATE KEY-----\\\\n*\\\\n*\\\\n ... \\\\n-----END PRIVATE KEY-----\\\\n\",\\n \"client_email\": \"[email protected]\",\\n \"client_id\": \"***\",\\n \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\\n \"token_uri\": \"https://oauth2.googleapis.com/token\",\\n \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\\n \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/connect%40project-dev.iam.gserviceaccount.com\",\\n \"universe_domain\": \"googleapis.com\"\\n}"
newlines	`\n`	`\\n`
newlines in private key	`\\n`	`\\\\n`

When I do kamal secret print it looks ok -- as in, it looks like the example i share above. My terminal displays it as properly tabbed JSON and the newlines in the private key are \n.

When I do kamal shell and echo it looks like this:

rails@178:/rails$ echo $GOOGLE_SERVICE_ACCOUNT_CONFIG {\n "type": "service_account",\n "project_id": "project-dev",\n "private_key_id": "******",\n "private_key": "-----BEGIN PRIVATE KEY-----\\n ... \\n-----END PRIVATE KEY-----\\n",\n "client_email": "[email protected]",\n "client_id": "******",\n "auth_uri": "https://accounts.google.com/o/oauth2/auth",\n "token_uri": "https://oauth2.googleapis.com/token",\n "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",\n "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/connect%40project-dev.iam.gserviceaccount.com",\n "universe_domain": "googleapis.com"\n}

Currently looking for a workaround. Got close with this:

doppler secrets -p project-dev -c prd get GOOGLE_SERVICE_ACCOUNT_CONFIG --plain | jq -c .

but this results in an RSA error. The extra \n are removed at the key level but the private key still has \\\\n in console.

Will post if I find something that works.

Happy dig/share more if I can get some guidance. Thanks!

Feb 12 '25 05:02 QuillyT

ok this worked:

GOOGLE_SERVICE_ACCOUNT_CONFIG=$(doppler secrets -p project-dev -c prd get GOOGLE_SERVICE_ACCOUNT_CONFIG --plain | jq -c . | sed 's/\\\\n/\\n/g')

Feb 12 '25 06:02 QuillyT

ok this worked:

GOOGLE_SERVICE_ACCOUNT_CONFIG=$(doppler secrets -p project-dev -c prd get GOOGLE_SERVICE_ACCOUNT_CONFIG --plain | jq -c . | sed 's/\\\\n/\\n/g')

I'm using AWS secrets manager and was running into a similar json problem to yours. My secrets are also in a JSON blob (as far as I can tell, that's all AWS supports). I tried a similar workaround, but I keep getting an error about jq not being found on my server. I verified that I installed jq on that machine, so I'm stumped about why kamal can't find it when it is fetching secrets. If I copy the "command" part of that statement and run it on my server, it works.

I'm trying something like this:

SECRETS=$(aws secretsmanager get-secret-value --secret-id my-secret | /usr/bin/jq '.SecretString | fromjson')

FWIW, I tried the built-in Kamal provider for AWS secrets manager, but that was giving me the JSON parse error.

Mar 15 '25 03:03 LeeHester

Secret that stores a JSON value being set in a way that causes JSON::ParserError in app

Context

DIFF