Secrets fail to load half the time

[evandam]

trafficstars

We're using secrets-init in combination with godotenv to source environment variables and then inject secrets into the environment.

We're seeing unexpected behavior where secrets-init does not set environment variables properly roughly half the time.

I also noticed that secrets-init does not overwrite an existing environment variable if it's already set. Is this by design or a bug?

❯ cat .env
ENCRYPTION_KEY=
AWS_REGION=us-west-2
AWS_SECRETS=arn:aws:secretsmanager:us-west-2:123456789012:secret:my-secret

❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:32
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:34
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:35
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:37
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:39
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

[evandam]

We also tried to find a pattern by trying to trigger rate limiting from Secrets Manager to see if it was related, but it doesn't look like it. It's very random.

❯ seq 1 50 | xargs -I% -P50 sh -c '{ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY; }'
ERRO[0002] failed to resolve secrets                     error="failed to get secret from AWS Secrets Manager: ProcessProviderExecutionError: error in credential_process\ncaused by: wait: no child processes"
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager

Edit: It seems that the issue is happening if the environment variable has a default value (ex: ENCRYPTION_KEY=), this will sporadically fail.

[Jujubasss]

I'm aiming to use the secrets-init too.. In order to implement some new tech here in the company we need to do some stress testing.. We have some namespaces with 2k+ pods, so i will keep one eye in this issue and will try to do some testing myself too.

[alexei-led]

@evandam it's not clear how to reproduce the problem. Can you provide a detailed scenario?

[evandam]

Hey @alexei-led, it's been a while but I believe the scripts in the initial message should be enough, unless you're saying you're not able to reproduce the issue? It's possible it was resolved incidentally but I'm not sure.

I believe the issue stemmed from the fact that if the secret in Secrets Manager is key/value pairs, it should override environment variables that may already be set, if you agree.

So take this example:

arn:aws:secretsmanager:us-west-2:123456789012:secret:foo-bar:

{
  "FOO": "foo",
  "BAR": "bar"
}

export FOO=""
export BAR=arn:aws:secretsmanager:us-west-2:123456789012:secret:foo-bar

secrets-init -- env | grep FOO
# NOTE: Sometimes FOO is the initial empty string
FOO=
# NOTE: Sometimes FOO is the secret value
FOO=foo

[alexei-led]

Unfortunately, this is the way Go cmd.Start() works when creating a new process.