Build cache not working in local mode

[cailloumajor]

Expected Behavior

Pushing an application to a device in local mode should take advantage of build layer caching.

Actual Behavior

Given the following multi-stage Dockerfile.template:

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian-node:14-buster-build AS builder

WORKDIR /usr/src/app

COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

COPY tsconfig.json ./
COPY src/*.ts src/
RUN yarn build


FROM balenalib/%%BALENA_MACHINE_NAME%%-debian-node:14-buster-run AS final

RUN set -eux; \
    install_packages \
        chromium \
        chromium-sandbox \
        dbus-x11 \
        gosu \
        libgl1-mesa-dri \
        tini \
        x11-xserver-utils \
        xinit \
        xserver-xorg-core \
        xserver-xorg-input-libinput \
        xserver-xorg-legacy \
        xserver-xorg-video-fbdev \
    ; \
    mkdir --mode=1777 /tmp/.X11-unix; \
    echo "allowed_users=anybody" > /etc/X11/Xwrapper.config; \
    useradd --user-group --system --create-home --no-log-init node-app; \
    usermod --groups tty node-app;
WORKDIR /usr/src/app

ENV NODE_ENV=production

COPY package.json yarn.lock ./
RUN set -eux; \
    yarn install --frozen-lockfile; \
    yarn cache clean

COPY --from=builder /usr/src/app/dist ./dist

ENTRYPOINT [ "tini", "--", "/usr/bin/entry.sh" ]

ENV DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket

COPY .xsession /home/node-app/
COPY start.sh /usr/local/bin/
CMD [ "/usr/local/bin/start.sh" ]

Each time I run balena push --noparent-check --nolive xxxxxxx.local, a complete build of the image is done, even if only start.sh file (which intervenes at the very end of the build process) is modified. I also tried to run the same command with no modification of the code, giving the same effect.

Using DEBUG=1 env var, the debug log just before the build starts shows:

[debug] original argv0="balena" argv=[/home/nono/balena-cli/balena,/snapshot/versioned-source/bin/balena,push,--noparent-check,--nolive,e7777e9.local] length=6
[debug] new argv=[/home/nono/balena-cli/balena,/snapshot/versioned-source/bin/balena,push,--noparent-check,--nolive,e7777e9.local] length=6
[Debug]   Using build source directory: . 
[Debug]   Pushing to local device: e7777e9.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://192.168.1.35:48484/ping
[Debug]   Checking device supervisor version: 12.6.8
[Info]    Starting build on device 192.168.1.35
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "."
[Info]    Creating default composition with source: "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://192.168.1.35:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Probing remote daemon for cache images
[Debug]   Using 1 on-device images for cache...
[Debug]   Starting builds...

I looked for a reason for the COPY package.json yarn.lock ./ instruction to issue a cache miss, and found that for each new build, the checksum of copied files (given by balena history --no-trunc <IMAGE ID>) is the same, in which case balena engine should hit the cache of the last build.

Specifications

• balena CLI version: balena-cli version "12.44.13" Node.js version "12.18.1"
• Cloud backend: openBalena or balenaCloud? N/A (local mode)
• Operating system version: Debian GNU/Linux bullseye (testing)
• 32/64 bit OS and processor: Linux 5.10.0-6-amd64, x86_64 processor
• Install method: zip package
• If npm install, Node.js and npm version: N/A

[cailloumajor]

In addition to being annoying to wait for the full build each time, it fills up the device storage.

[ToolKami]

@cailloumajor running into same issue, did you manage to resolve this?

[cailloumajor]

@ToolKami no