build-push-action Local cache remains empty despite cache-to and cache-from configuration

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

[X] The documentation does not mention anything about my problem
[ ] There are no open or closed issues that are related to my problem

Description

I'm running Docker on a Self Hosted Github Actions Runner, and I've configured a local cache path at ~/.cache/docker-ci, as well as specifying it in build-push-action fields.

However, the cache remains empty after each build

Expected behaviour

The cache should be populated and used by Docker to speedup the future builds.

Actual behaviour

The cache remains empty:

$ du -h ~/.cache/docker-ci
4.0K    /home/mtarral/.cache/docker-ci/

Repository URL

No response

Workflow run URL

No response

YAML workflow

jobs:
  build:
    runs-on: self-hosted
    strategy:
      matrix:
        baseimage: ['python:3.12.1-bullseye', 'pypy:3.10-7-bullseye']
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: 3.11

      - uses: docker/setup-buildx-action@v3

      - uses: docker/[email protected]
        with:
          context: .
          file: ./Dockerfile.dev
          load: true
          cache-from: type=local,src=~/.cache/docker-ci
          cache-to: type=local,dest=~/.cache/docker-ci,mode=max

Workflow logs

#39 exporting cache to client directory
#39 preparing build cache for export
#39 writing layer sha256:09e2bc8a597c33b54cccaf52f2e21798e2e0df79ab6cb33d3b1dfd4b33a57512
#39 writing layer sha256:09e2bc8a597c33b54cccaf52f2e21798e2e0df79ab6cb33d3b1dfd4b33a57512 0.4s done
#39 writing layer sha256:0d8ca508c70e4a47dd31c967eb8515ec83e26d200c3ee198531ac77278921875
#39 writing layer sha256:0d8ca508c70e4a47dd31c967eb8515ec83e26d200c3ee198531ac77278921875 0.6s done
#39 writing layer sha256:10a7af5bb9f35aa9e58f3026537e84d1ab85ddc302387c0e78b7fb5a25dcf7ca
#39 writing layer sha256:10a7af5bb9f35aa9e58f3026537e84d1ab85ddc302387c0e78b7fb5a25dcf7ca 0.5s done
#39 writing layer sha256:358218251c64aaed2cde2fa39eed7547cd6dfb1fdc5f11a4cced3a3f07d2d949
#39 writing layer sha256:358218251c64aaed2cde2fa39eed7547cd6dfb1fdc5f11a4cced3a3f07d2d949 0.0s done
#39 writing layer sha256:3762059b197e964a4543a56700f73f4a28279a1175b3029c0091ce39e054fbf0
#39 writing layer sha256:3762059b197e964a4543a56700f73f4a28279a1175b3029c0091ce39e054fbf0 0.3s done
#39 writing layer sha256:378eaea555235e1b6cd8849b4490f1c6812b2d657ecf2f42633d301dfdc01c1e
#39 writing layer sha256:378eaea555235e1b6cd8849b4490f1c6812b2d657ecf2f42633d301dfdc01c1e 0.5s done
#39 writing layer sha256:40b118dff4300bb1b3256a3aff2203c330b817609e0d0858c1864379dc0f59bb
#39 writing layer sha256:40b118dff4300bb1b3256a3aff2203c330b817609e0d0858c1864379dc0f59bb done
#39 writing layer sha256:425f77304b3dd0d7d8103d5628e58d270403d41201e204f54ec4063e42b1eef0 done
#39 writing layer sha256:43c7d862cba465d342dbf73dca7caf5e04c2ec7b374c918ec26f305e2ba3f78f
#39 writing layer sha256:43c7d862cba465d342dbf73dca7caf5e04c2ec7b374c918ec26f305e2ba3f78f 0.4s done
#39 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#39 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#39 writing layer sha256:5b62a7b1c0aaa1470515d89fc7eeeebaf8c21dbccb80516929436c12c2dddd30
#39 writing layer sha256:5b62a7b1c0aaa1470515d89fc7eeeebaf8c21dbccb80516929436c12c2dddd30 6.9s done
#39 writing layer sha256:5c357c4a41dc4391be81ff470ca10d7683587987b66b09eb89e2a227a75ef63c
#39 writing layer sha256:5c357c4a41dc4391be81ff470ca10d7683587987b66b09eb89e2a227a75ef63c done
#39 writing layer sha256:65673bc22dbb7a3930bb78149565f1d7f551e16f450c46f8999e505938488c88
#39 writing layer sha256:65673bc22dbb7a3930bb78149565f1d7f551e16f450c46f8999e505938488c88 0.5s done
#39 writing layer sha256:65b7f93b3e993c7cf0973dc9e0a75a80b163cc5ec08624659a139df5d6466573
#39 writing layer sha256:65b7f93b3e993c7cf0973dc9e0a75a80b163cc5ec08624659a139df5d6466573 0.0s done
#39 writing layer sha256:6851ce4cf38f7b24ac921524d92063e4e0d9c9890b720ce734a26ca6bd33d13f done
#39 writing layer sha256:857b20b5b33decba2fc85810e148859e80b3cb5312629983e8a88f4031640db2 0.0s done
#39 writing layer sha256:9468f2c57f0a3d46cc92965d2afbc39a5b2efe34687fe3f9e8e62be7b96ac30d done
#39 writing layer sha256:95530d5aec62b45b5926bfa2e457f6ebd50a2784e47682ef1f1bd501c65bc151 done
#39 writing layer sha256:9bcbd0d1a254909d3f62727c0254a66eca78f8ecdc5f567af3b96e6903aee041 done
#39 writing layer sha256:a12cc1944b57bdc91343b600de5bc9f06f5591c3738f239055e2b5c12c654e64
#39 writing layer sha256:a12cc1944b57bdc91343b600de5bc9f06f5591c3738f239055e2b5c12c654e64 0.4s done
#39 writing layer sha256:a1bbf2983642e080d705d575c1da8d4d8c35507576d88e44979b5c6229573d40
#39 writing layer sha256:a1bbf2983642e080d705d575c1da8d4d8c35507576d88e44979b5c6229573d40 0.1s done
#39 writing layer sha256:a6d11da8a0c75ddf4be5e158ffe4e31edeb48afaa8391dbe3e3de00197a8654b
#39 writing layer sha256:a6d11da8a0c75ddf4be5e158ffe4e31edeb48afaa8391dbe3e3de00197a8654b done
#39 writing layer sha256:a89a8335d6e112ae1cc6ac1a30abcd4ddae5d361ebf96ce30f664a26eae007d5 done
#39 writing layer sha256:b74f8c19aebcbde0ec5866826963e8f2df033d89e0edc459db14a5eb0d98c871 done
#39 writing layer sha256:bf34e9f18884bdeae0349bfb408d53c21122f06c19c9393216e5a9b5f49450a4
#39 writing layer sha256:bf34e9f18884bdeae0349bfb408d53c21122f06c19c9393216e5a9b5f49450a4 0.3s done
#39 writing layer sha256:cbd67245b7c92987cf5150f71c5dfa0d0f1da3bcda17a12ba6b9768f2ed7df28
#39 writing layer sha256:cbd67245b7c92987cf5150f71c5dfa0d0f1da3bcda17a12ba6b9768f2ed7df28 done
#39 writing layer sha256:d0209a266bb24310efc230a2cedc8c753df202b1367d6b917b3a6febaaa225fd
#39 writing layer sha256:d0209a266bb24310efc230a2cedc8c753df202b1367d6b917b3a6febaaa225fd 1.5s done
#39 writing layer sha256:d3c71e743257d3241d56524f77910aa1d23756a4b88594923c91881e2ade9237
#39 writing layer sha256:d3c71e743257d3241d56524f77910aa1d23756a4b88594923c91881e2ade9237 done
#39 writing layer sha256:d553b4c0e78230b547c734037fe72ecdc7c470af2debbba73a5c84f3b5cfdf8c 0.0s done
#39 writing layer sha256:e4e7d5dca38c9ee4e297c10c17768b3a9d79cac432c28e86c0a23414292555ae done
#39 writing layer sha256:e5971d2b8b74f6f831bce1c391f6f2a8244b0fa148f651960d092b17069f16db done
#39 writing layer sha256:e60118d7061c7d2500e03283660c7686ba3c90cd33efc8dc8e37e34e1e5a18e2 0.0s done
#39 writing layer sha256:e83ecebbd847d9388fba25170ed8276f3595c657b4465fffbe0d343426ff35de
#39 writing layer sha256:e83ecebbd847d9388fba25170ed8276f3595c657b4465fffbe0d343426ff35de 9.0s done
#39 writing layer sha256:f0c626cc4a171db377e7bf695b15fe45dc8766a931ca31686a8bf39e6c038040
#39 writing layer sha256:f0c626cc4a171db377e7bf695b15fe45dc8766a931ca31686a8bf39e6c038040 0.2s done
#39 preparing build cache for export 127.2s done
#39 writing config sha256:ba37f44cf476e7652f353135d3633d7f7baa1c8d1f2e1c86d260849a9857f737 done
#39 writing cache manifest sha256:02940af896a9755db5a4694122702d3da874755244baa11162bcfcd84aefb310 done
#39 DONE 127.2s
WARNING: local cache import at ~/.cache/docker-ci not found due to err: could not read ~/.cache/docker-ci/index.json: open ~/.cache/docker-ci/index.json: no such file or directory
ImageID
  sha256:534bd6fd20266b532a1a00fcdea9ab666f20dc41fecc957301e69412057ca6c1
Digest
  sha256:8807551c0a9cbe46237680bc600b4522f5fb43fa3d9ccfb6811618307de7688d

BuildKit logs

No response

Additional info

Related closed issue that I found:

https://github.com/docker/buildx/issues/206

I can't share the repository as it's a private project.

Feb 13 '24 15:02 Wenzel

Having the same issue. I'm trying all of the variously proposed options to get my github action cache to persist across multiple runners, cannot figure it out. So I tried running the build 2 times in the same runner, both times getting this issue where

WARNING: local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory

Run: https://github.com/mozilla/addons-server/actions/runs/8019019347/job/21906086963?pr=21893

Feb 23 '24 12:02 KevinMind

You need to set the actions/cache step in our workflow otherwise cache is not retained across runs. See our docs: https://docs.docker.com/build/ci/github-actions/cache/#local-cache

Tried in this repo through https://github.com/docker/build-push-action/pull/1066 and looks good to me:

first run (no index.json as expected): https://github.com/docker/build-push-action/actions/runs/8082221151/job/22082556076#step:7:243
second run (cache loaded): https://github.com/docker/build-push-action/actions/runs/8082268470/job/22082718550#step:7:170

Also you should consider using the gha cache backend which is more efficient: https://docs.docker.com/build/ci/github-actions/cache/#github-cache

Feb 28 '24 14:02 crazy-max

Thanks for testing and pointing to the docs @crazy-max.

I tried your setup, and the cache works for me (replacing /tmp/.buildx-cache by /tmp/.docker-cache)

$ du -h /tmp/.docker-cache/
1.6G    /tmp/.docker-cache/blobs/sha256
1.6G    /tmp/.docker-cache/blobs
4.0K    /tmp/.docker-cache/ingest
1.6G    /tmp/.docker-cache/

But as soon as I replace this path with ~/.cache/docker-ci, the caching doesn't work anymore. That folder remains empty.

      - name: Cache Docker layers
        uses: actions/cache@v4
        with:
          path: ~/.cache/docker-ci
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

I tried to remove this folder, and let the workflow create it, but it didn't work either. So I'm puzzled as to what really happens, and i'm forced to use a path starting with /tmp here.

Also you should consider using the gha cache backend which is more efficient: https://docs.docker.com/build/ci/github-actions/cache/#github-cache

That solution would be suitable if the cache content wasn't sensitive or private, which is the case here.

Feb 29 '24 14:02 Wenzel

path: ~/.cache/docker-ci

Don't think actions/cache@v4 expands ~. You should use an absolute dir.

Feb 29 '24 15:02 crazy-max