apk upgrade Alpine 3.12(+) fails in kaniko build

[Frettarix]

Hi,

We run container on a Kubernetes environment and run pipelines for the automated build via the GitlabRunner (using Kaniko to build the image). We often use alpine:latest and add layers on top of this, but our latest image update failed.

INFO[0005] RUN apk update && apk add curl && apk --no-cache upgrade INFO[0005] cmd: /bin/sh INFO[0005] args: [-c apk update && apk add curl && apk --no-cache upgrade] fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz v3.12.0-57-g76596d9eb3 [http://dl-cdn.alpinelinux.org/alpine/v3.12/main] v3.12.0-59-gf0047a0abb [http://dl-cdn.alpinelinux.org/alpine/v3.12/community] OK: 12732 distinct packages available (1/4) Installing ca-certificates (20191127-r3) (2/4) Installing nghttp2-libs (1.41.0-r0) (3/4) Installing libcurl (7.69.1-r0) (4/4) Installing curl (7.69.1-r0) Executing busybox-1.31.1-r16.trigger Executing ca-certificates-20191127-r3.trigger OK: 7 MiB in 18 packages fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz (1/2) Upgrading alpine-baselayout (3.2.0-r6 -> 3.2.0-r7) Executing alpine-baselayout-3.2.0-r7.pre-upgrade rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..data': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/token': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/namespace': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2020_06_10_13_34_23.122167960/namespace': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2020_06_10_13_34_23.122167960/ca.crt': Read-only file system rm: can't remove '/var/run/secrets/kubernetes.io/serviceaccount/..2020_06_10_13_34_23.122167960/token': Read-only file system Executing alpine-baselayout-3.2.0-r7.post-upgrade ERROR: alpine-baselayout-3.2.0-r7: failed to rename var/.apk.f752bb51c942c7b3b4e0cf24875e21be9cdcd4595d8db384 to var/run. (2/2) Upgrading ca-certificates-bundle (20191127-r2 -> 20191127-r3) Executing busybox-1.31.1-r16.trigger Executing ca-certificates-20191127-r3.trigger 1 error; 7 MiB in 18 packages error building image: error building stage: waiting for process to exit: exit status 1 ERROR: Job failed: command terminated with exit code 1

We started pinpointing the issue by changing the base image (alpine:latest) (which was the only thing that may have changed).

It seems that 3.12 and up 'apk upgrade' will try to alter /var/run/secrets which is not allowed by compliancy. The Kaniko image is running as root and has always been able to build in previous versions.

Any idea what we could do to allow the use of alpine;latest again for our container images?

[andrevdh]

Can confirm this issue is also affecting our pipelines which are based on alpine:3.12 or postgres:12.3-alpine ... not sure why apk upgrade needs to remove /var/run/secrets/* from the image as part of the upgrade process

[andrevdh]

We're going to run some tests using apk upgrade --ignore alpine-baselayout and see how that plays out ... it seems to remove the file removal bits in the upgrade process

[ncopa]

Can confirm this issue is also affecting our pipelines which are based on alpine:3.12 or postgres:12.3-alpine ... not sure why apk upgrade needs to remove /var/run/secrets/* from the image as part of the upgrade process

Apparently this was introduced by https://gitlab.alpinelinux.org/alpine/aports/-/commit/e5ba0e9172a83480b7c4e612b4aebd334b8ecfb9 to fix https://gitlab.alpinelinux.org/alpine/aports/-/issues/8029

But I think current alpine-baselayout is broken in at least two ways.

• [ -d /var/run ] returns true even if it already is a symlink
• apparently the package has a /var/run -> /run symlink while it should be a relative link /var/run -> ../run

This needs to be fixed.

[rufreakde]

https://gitlab.alpinelinux.org/alpine/aports/-/issues/8029 Seems still to be open I guess needs some bumps