azure-keyvault-env executable appears to only work in Alpine based images

[theseion]

Components and versions [ ] Controller, version: x.x.x (docker image tag) [x] Env-Injector (webhook), version: 1.3.0 (docker image tag) [x] azure-keyvault-env

The azure-keyvault-env executable fails to perform the network request to KeyVault in images that are not Alpine based. I noticed this first while using a postgres:11 image and then discovered that injection fails with Debian and Ubuntu images in general. I assume that the issue occurs because the executable is built on Alpine and, apparently, there appears to be a subtle difference in the network stack.

To Reproduce Use the tutorial steps but use a Debian or Ubuntu image as the target, not the pre-supplied test image.

Expected behavior The executable should work, regardless of the OS of the container. Note that it probably will not be enough to just build the executable for a couple of different distros, as the containers in a cluster can use a bunch of different distros and the Helm chart can only support one choice of distro for the env image.

Logs The logs always show the same pattern, where the command times out (authorization etc. works fine).

I0917 07:21:10.491765       1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv"
I0917 07:21:10.491881       1 main.go:176] "azure key vault env injector initializing"
I0917 07:21:10.492237       1 main.go:245] "found original container command" cmd="/bin/sh" args=[/bin/sh -c /usr/bin/env; sleep 30]
I0917 07:21:10.492286       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/foo/test-6bd5cdb549-8dl5x?secret=akv2k8s-test"
I0917 07:21:10.536002       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/foo/test-6bd5cdb549-8dl5x?secret=akv2k8s-test"
I0917 07:21:10.536363       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/foo/test-6bd5cdb549-8dl5x"
I0917 07:21:10.566988       1 authentication.go:167] "successfully received oauth token"
E0917 07:21:40.588373       1 main.go:334] "failed to read secret from azure key vault" err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" azurekeyvaultsecret="foo/test-password"

[theseion]

To test my hypothesis I rebuilt the env image with make image-vaultenv after modifying the Dockerfile to use debian:buster as the value for BASE_ALPINE, pushed that image to my own registry and updated the Helm deployment to use that image.

As it turns out, this changes nothing. The executable still works in Alpine based images but not in Debian based images, even though the executable was built using Debian buster.

Also: using postgres:11-alpine instead of postgres:11 fixes the problem for me. Unfortunately, there are images that I can't switch to Alpine...

[Haavare]

Hi, The executable is built in golang:1.16.5 which is a debian. BASE_ALPINE is the runtime image of the components. I will try to reproduce

[theseion]

Let me know if I can provide any more information.

[Haavare]

Progressing i bit slowly, but reproducing was not hard. Will see if I can narrow it down

[Firgeis]

I am getting the same issue with image https://hub.docker.com/r/bitnami/pgbouncer/ :

I0111 23:25:21.591919       1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv"
I0111 23:25:21.592014       1 main.go:176] "azure key vault env injector initializing"
I0111 23:25:21.592263       1 main.go:245] "found original container command" cmd="/opt/bitnami/scripts/pgbouncer/entrypoint.sh" args=[/opt/bitnami/scripts/pgbouncer/entrypoint.sh /opt/bitnami/scripts/pgbouncer/run.sh]
I0111 23:25:21.592338       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.security.svc:80/auth/services/services-ops-pgbouncer-f4674b79f-b5zlf?secret=akv2k8s-services-ops-pgbouncer"
I0111 23:25:21.755178       1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.security.svc:80/auth/services/services-ops-pgbouncer-f4674b79f-b5zlf?secret=akv2k8s-services-ops-pgbouncer"
I0111 23:25:21.755765       1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.security.svc:9443/auth/services/services-ops-pgbouncer-f4674b79f-b5zlf"
I0111 23:25:21.870332       1 authentication.go:167] "successfully received oauth token"
E0111 23:25:51.924907       1 main.go:334] "failed to read secret from azure key vault" err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" azurekeyvaultsecret="services/pg-ds-db"

[BlauerPulli]

i also faced this and its really kind of a show-stopper. after first researches i'm a bit confused because:

1. alpine is a lightweight linux so i would expect such problems more in alpine than on debian
2. ldd says the azure-keyvault-env is a full static binary, so no dependencies are missing
3. its generally executable in debian as it only fails in a specific part of the code (within the function in cmd/azure-keyvault-env/main.go:104 getSecretFromKeyVault). Need to dig deeper in the code and its dependencies here if maybe somewhere are os specific things ongoing, but i'm not an golang specialist at all.

Generally it would nice to hear a living sign from maintainers/contributors.. Seems they're in vacation since almost 3 months?!(no merged PR since then). Would be sad if this project fall asleep.

[theseion]

@BlauerPulli That's basically what I found too. My guess, without actually looking at the code, is that there's a difference between musl and glibc. Something along the lines of this SO question: https://stackoverflow.com/questions/54524785/how-to-build-a-go-executable-that-doesnt-link-to-musl-libc.

[BlauerPulli]

@theseion I tried different things: to build the binary in an ubuntu env, with CGO_ENABLED=0 GOARCH=amd64 options and additionally to inject a secret into a pod based on a debian image where musl package is installed on top- nothing helped.

[theseion]

I don't really have time to dig into this but I tried to track down where the call happens. This is the line where the call is initiated: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/blob/bb751aef3405ab1b0118576f73635667e8481a10/pkg/azure/keyvault/client/service.go#L75. The client is an object from azure autorest: https://github.com/Azure/azure-sdk-for-go/tree/main/services/keyvault. autorest uses go base packages net and http: https://github.com/Azure/go-autorest/blob/b3899c1057425994796c92293e931f334af63b4e/autorest/sender.go#L129.

So maybe we could create a test with that version of autorest directly, or copy that part of the code from autorest and try reproduce the error.

[theseion]

I give up. I've played around with the keyvault package, but since it's an old package (2016-10-01) there's barely any documentation and getting authentication set up is a nightmare (there's no interactive browser authentication, so I'd have to set up additional resources etc.).

Seeing that it's an old package, maybe the problem would be resolved simply by moving to the current SDK (obviously that would be a lot more work than it sounds but maybe its enough for testing).

[tspearconquest]

The 1.5.0 beta releases, I think have switched to a newer SDK. Give one of those a try.

[theseion]

Thanks @tspearconquest. Unfortunately, the project has been canned, so I'm not using azure-keyvault-env at the moment.