Rewriting timestamps seems to skip cached layers

[jan-kiszka]

While making the kas (https://github.com/siemens/kas) container images reproducible, I noticed that cached layer from previous builds did not seem to get their timestamps updated if they are used in a successive build with different SOURCE_DATE_EPOCH. We are avoiding this issue now by not persisting the layer cache on github builds and disabling caching completely on local validation builds - which is rather unfortunate as it includes the package cache.

Is this a known issue?

[AkihiroSuda]

force-compression may work as a workaround? (didn't confirm)

[jan-kiszka]

Probably, though not really applicable in our case. We have two images with a common base which would then be duplicated.

[tonistiigi]

Tentatively for v0.13.1 but needs to be investigated further

[AkihiroSuda]

Needs a minimal reproducer.

I cannot repro the issue with the following script:

#!/bin/bash
set -eux

build() {
        epoch="$1"
        dest="$2"
        rm -rf "$dest"
        buildctl build \
                --frontend dockerfile.v0 \
                --local dockerfile=. --local context=. \
                --opt build-arg:SOURCE_DATE_EPOCH="$epoch" \
                --opt platform=linux/amd64 \
                --output type=oci,dest="$dest",tar=false,rewrite-timestamp=true
}

cat <<EOF >Dockerfile
FROM busybox:1.36.1@sha256:650fd573e056b679a5110a70aabeb01e26b76e545ec4b9c70a9523f2dfaf18c6
RUN echo f0 >/f0
EOF
build 0 out.0

cat <<EOF >>Dockerfile
RUN echo f1 >/f1
EOF
build 1 out.1

[thompson-shaun]

Would you be able to provide any insight/reproduction guidance @jan-kiszka ?