cache
cache copied to clipboard
actions
Docker caching
Should we try to use this cache action to cache docker layers, doing trickery with docker save and docker load, or are you working on a different path for Docker caching?
You're welcome to use this for caching docker layers! If you get something figured out we can add it as an example usage of this action.
are you working on a different path for Docker caching?
Nothing is in scope right now for v1 of this action that's specific to the Docker scenario, but it might already work.
Related: https://github.com/actions/toolkit/issues/197
I tried but I hit the file limit, discussed at https://github.com/actions/cache/issues/6
Since docker is now natively integrated in Github actions, I think it would be nice if there was an option to enable docker caching globally, like CircleCI does.
@tuler also tried to cache with a tarball on https://github.com/actions/cache/issues/33. It seems to be a good solution.
https://github.com/actions/cache/pull/37 also was opened.
Opened PR https://github.com/actions/cache/pull/37 with an example for Docker layer caching.
@steebchen I agree. Docker caching should be natively supported, not through this action.
The cache file limit has been increased from 400 MB to 2 GB, so this may be more viable than previously due to docker layers being larger than the previous limit.
@steebchen I agree. Docker caching should be natively supported, not through this action.
Seems like a lot of people agree to that statement. Where is the proper place to post such a request? In the runner repo? Perhaps someone in the inner circle should post it as a feature request in the appropriate place and post a link here?
@DannyBen See https://github.com/actions/cache/issues/81 where we're tracking Docker caching of actions
👋 Hi all, right now we are focusing on other priorities and there are no updates for docker caching. We appreciate your feedback and revisit priorities based on user feedback, so please continue sending us your input.
What's bothering me most is that all Github Actions (which are based on Docker) are always rebuild from scratch, in every single action. I've never worked in projects as big as GitHub, but I can't imagine how caching is not the solution here – just for the reason that it costs bandwidth to fetch the images each time. I'd also be happy to upgrade to a more expensive plan or pay for the cache storage costs.
(edit: I just realised I already commented in this thread... :smile:)
Working example for me using buildx here: https://github.com/actions/cache/issues/260#issuecomment-638469605
My ci processes use several pre-built docker images, some of them are very large - 3.5GB! github actions pulls these images again and again on every run burning a lot of time and cpu. A built in cache thats got a reasonable expiry behind each repo is really required here. Sooo much dealing with github actions seems to be spent working around paradigm/api failures or missing features. This is a big one, please give us some better options.
Hello, I am wondering if this issue has bubbled back up on the action/cache's priority list? I am wondering if there is a difficult technical challenge blocking this feature, as on its surface it seems like it would be a huge benefit to both users running actions and reduce load on GH infrastructure.
@mikelax Would https://github.com/satackey/action-docker-layer-caching work for your case? It's built on top of this action.
@mikelax Would https://github.com/satackey/action-docker-layer-caching work for your case? It's built on top of this action.
This linked action says it will use following method to take care of caching:
This GitHub Action uses the docker save / docker load command and the @actions/cache library.
While ago, @dtinth published an article Caching Docker builds in GitHub Actions: Which approach is the fastest? 🤔 A research. (https://dev.to/dtinth/caching-docker-builds-in-github-actions-which-approach-is-the-fastest-a-research-18ei). He found the docker save/load method is far from the best solution. Worth taking a look whether you are trying to cut some seconds.
I would also love to see a solution for this. I've had a few actions spend hundreds of billable minutes trying to pull docker images, and its costing a significant amount of money. I'd much rather have the docker layers cached.
But when I try to cache the docker folder, I get this warning.
Warning: EACCES: permission denied, scandir '/var/lib/docker'
Hi, I wrote the article mentioned above which has been outdated and superseded by another Evil Martians’ article.
TL;DR — Docker has published an official action docker/build-push-action which has an option to make Docker Buildx use GitHub Cache API directly as a cache storage, skipping the need to use a local-filesystem-based cache such as actions/cache entirely. With this settings my build time dropped from 5 minutes to 1 minute.
cache-from: type=gha
cache-to: type=gha,mode=max
While the setup-buildx suggestion works when you're building your own docker images, AFAICT it doesn't work when you're using docker images as container services. It would be great if this was supported.
The setup-buildx line of actions is also a beast to try to use if you have your test services and container under test setup through docker-compose. There is no obvious way to get the docker provided actions to work with the cache.
The cache option on docker/build-push-action works great for building images.
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-upload-docker-image:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
packages: write
outputs:
docker-tag: ${{ steps.meta.outputs.tags }}
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Log in to the Container registry
uses: docker/login-action@v1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v3
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
push: true
file: Dockerfile.prod
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: |
type=registry,ref=${{ steps.meta.outputs.tags }}
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:master
cache-to: type=inline
However, I have a use case for running tests in a dockerized setup.
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 10
needs: [build-and-upload-docker-image]
steps:
- uses: actions/checkout@v2
- name: Log in to the Container registry
uses: docker/login-action@v1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.PACKAGE_ACCESS_TOKEN }}
- name: Run tests
run: docker-compose run web yarn test
Here every run pulls dependencies defined in docker-compose (e.g. postgres, redis, etc). Could caching of those be supported natively or by an action?
