action-docker-layer-caching
action-docker-layer-caching copied to clipboard
satackey
Add a filter parameter - to select for the images to cache
Is your feature request related to a problem? Please describe. Not all images need caching, and can take up valuable room. Some images are loaded before the cache is loaded such as the images in services. So excluding those images would help reduce the size and speed up the save/load process.
Describe the solution you'd like
When doing docker save $(docker images -q) you can provide a list of the image ID's to save.
If we could provide a filter at that point, it would mean we were only saving the images that we needed and so instead of this list:
REPOSITORY TAG IMAGE ID CREATED SIZE
codacy/codacy-prospector 2.3.9 2ed875f0c09d 2 days ago 267MB
codacy/codacy-analysis-cli latest 9d412c689197 2 days ago 421MB
snyk/snyk maven-3-jdk-11 882d05531c51 3 days ago 734MB
codacy/codacy-rubocop 5.1.6 88f504735018 7 days ago 166MB
codacy/codacy-tsqllint 1.2.4 92b7ddf6589f 7 days ago 91.8MB
codacy/codacy-pylint-python3 2.2.16 504237065f5b 7 days ago 161MB
api 11 413eb2357edb 9 days ago 512MB
codacy/codacy-bandit 2.3.1 5c95af379274 12 days ago 264MB
codacy/codacy-pmd 3.11.0 bb50d6e2a005 2 weeks ago 162MB
tomcat 8.5-jdk11-corretto ae7b0918b39c 4 weeks ago 473MB
codacy/codacy-remark-lint 2.5.59 ef1454679208 6 weeks ago 155MB
codacy/codacy-bundler-audit 4.2.12 8d9828916ae8 2 months ago 86MB
redis 5-alpine caa6655434a5 2 months ago 29.7MB
codacy/codacy-spotbugs 1.10.5 999b0c7e7945 2 months ago 128MB
amazonlinux xray b88042e00023 3 months ago 395MB
codacy/codacy-checkstyle 2.3.0 5b7b8322d896 4 months ago 118MB
codacy/codacy-hadolint 1.5.2 04e2b64ed099 4 months ago 9.31MB
codacy/codacy-metrics-pmd 0.2.2 8b5299fc6999 4 months ago 111MB
codacy/codacy-shellcheck 2.4.1 509fffd67432 4 months ago 33.5MB
codacy/codacy-sqlint 2.3.2 f21a8ce8c730 4 months ago 148MB
codacy/codacy-jackson-linter 5.2.2 d06c7b65bd22 4 months ago 104MB
codacy/codacy-brakeman 1.3.2 f95377ddb330 4 months ago 146MB
codacy/codacy-pylint 3.2.1 582388b48c67 4 months ago 242MB
codacy/codacy-codenarc 0.4.2 88739f96d424 4 months ago 111MB
mysql 5.6 c580203d8753 4 months ago 302MB
codacy/codacy-metrics-cloc 0.2.4 0f47e48065ab 7 months ago 197MB
amazon/amazon-ecs-local-container-endpoints latest b005329f50b1 7 months ago 180MB
codacy/codacy-duplication-pmdcpd 2.0.145 e121e0516f79 11 months ago 132MB
codacy/codacy-metrics-rubocop 0.2.2 ee82a4e0ba6a 11 months ago 269MB
codacy/codacy-metrics-radon 0.1.66 19f7a80d4001 11 months ago 154MB
codacy/codacy-duplication-flay 2.0.154 ba95f1e54cfa 11 months ago 166MB
codacy/codacy-pmdjava 2.0.0-pmdlegacy.57fdbf2 dee6d8e1c0e5 20 months ago 231MB
boxfuse/flyway 5.2.4-alpine df49c5fb49bb 2 years ago 115MB
memcached 1.5.10-alpine fb0df03da449 2 years ago 8.97MB
A filter would be added at this point:
docker save $(docker images -q --filter=reference='codacy/*')
and we would get this list instead:
REPOSITORY TAG IMAGE ID CREATED SIZE
codacy/codacy-prospector 2.3.9 2ed875f0c09d 2 days ago 267MB
codacy/codacy-analysis-cli latest 9d412c689197 2 days ago 421MB
codacy/codacy-rubocop 5.1.6 88f504735018 7 days ago 166MB
codacy/codacy-tsqllint 1.2.4 92b7ddf6589f 7 days ago 91.8MB
codacy/codacy-pylint-python3 2.2.16 504237065f5b 7 days ago 161MB
codacy/codacy-bandit 2.3.1 5c95af379274 12 days ago 264MB
codacy/codacy-pmd 3.11.0 bb50d6e2a005 2 weeks ago 162MB
codacy/codacy-remark-lint 2.5.59 ef1454679208 6 weeks ago 155MB
codacy/codacy-bundler-audit 4.2.12 8d9828916ae8 2 months ago 86MB
codacy/codacy-spotbugs 1.10.5 999b0c7e7945 2 months ago 128MB
codacy/codacy-checkstyle 2.3.0 5b7b8322d896 4 months ago 118MB
codacy/codacy-hadolint 1.5.2 04e2b64ed099 4 months ago 9.31MB
codacy/codacy-metrics-pmd 0.2.2 8b5299fc6999 4 months ago 111MB
codacy/codacy-shellcheck 2.4.1 509fffd67432 4 months ago 33.5MB
codacy/codacy-sqlint 2.3.2 f21a8ce8c730 4 months ago 148MB
codacy/codacy-jackson-linter 5.2.2 d06c7b65bd22 4 months ago 104MB
codacy/codacy-brakeman 1.3.2 f95377ddb330 4 months ago 146MB
codacy/codacy-pylint 3.2.1 582388b48c67 4 months ago 242MB
codacy/codacy-codenarc 0.4.2 88739f96d424 4 months ago 111MB
codacy/codacy-metrics-cloc 0.2.4 0f47e48065ab 7 months ago 197MB
codacy/codacy-duplication-pmdcpd 2.0.145 e121e0516f79 11 months ago 132MB
codacy/codacy-metrics-rubocop 0.2.2 ee82a4e0ba6a 11 months ago 269MB
codacy/codacy-metrics-radon 0.1.66 19f7a80d4001 11 months ago 154MB
codacy/codacy-duplication-flay 2.0.154 ba95f1e54cfa 11 months ago 166MB
codacy/codacy-pmdjava 2.0.0-pmdlegacy.57fdbf2 dee6d8e1c0e5 20 months ago 231MB
as in at this part of the code change it to this:
import * as exec from 'actions-exec-listener'
import * as core from '@actions/core'
export class ImageDetector {
async getExistingImages(): Promise<string[]> {
const _filter = core.getInput(`filter`)
const filter = _filter ? `--filter=${_filter}` : ''
const existingSet = new Set<string>([])
const ids = (await exec.exec(`docker image ls -q ${filter}`, [], { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``)
const repotags = (await exec.exec(`docker`, `image ls --format {{.Repository}}:{{.Tag}} ${filter} --filter=dangling=false`.split(' '), { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``);
core.debug(JSON.stringify({ log: "getExistingImages", ids, repotags }));
([...ids, ...repotags]).forEach(image => existingSet.add(image))
core.debug(JSON.stringify({ existingSet }))
return Array.from(existingSet)
}
async getImagesShouldSave(alreadRegisteredImages: string[]): Promise<string[]> {
const resultSet = new Set(await this.getExistingImages())
alreadRegisteredImages.forEach(image => resultSet.delete(image))
return Array.from(resultSet)
}
}
this would be cool, at the moment i'm caching many images that really dont need to be (like mysql etc), but OTOH I really want to cache the images that are built locally.. (I'm running this as docker-compose)
Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏
Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏
I am using a fork where it's working in production.
https://github.com/Broadshield/action-docker-layer-caching
@jamie-wearsafe Thanks! Using your fork for now, hopefully the filter input can be merged into this project.
For anyone curious on how to use this here is an example:
- name: Cache docker images
uses: Broadshield/action-docker-layer-caching@main
continue-on-error: true
with:
filter: reference=my_custom_images_prefix*
@jamie-wearsafe Thanks! Using your fork for now, hopefully the
filterinput can be merged into this project.For anyone curious on how to use this here is an example:
- name: Cache docker images uses: Broadshield/action-docker-layer-caching@main continue-on-error: true with: filter: reference=my_custom_images_prefix*
Ahh sorry I didn't provide an example!
@jamie-wearsafe Great work, however im seeing this:
Post job cleanup.
/usr/bin/docker image ls --format={{.ID}} {{.Repository}}:{{.Tag}} --filter=dangling=false --filter=reference='openmined/*'
There is no image to save.
However when using tunshell.com to debug the CI process I can run that exact command and it outputs several hashes...
https://github.com/OpenMined/PySyft/runs/3511338358?check_suite_focus=true
@jamie-wearsafe your fork worked great for me. I needed to cache a single image to be restored by a job later in my workflow. Without the filter, the cache would load all images from previous runs (because they are uniquely tagged).
- name: Set Image Tag
run: echo "GITHUB_SHA_SHORT=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
- name: Cache Container Image
uses: Broadshield/action-docker-layer-caching@main
with:
filter: reference=account/imagename:${{ env.GITHUB_SHA_SHORT }}
@madhavajay i know its been a long time but do you need that issue addressed?
Hi @jamie-wearsafe yeah that would be great, I am currently just using cache all and my guess is the cache is getting regularly evicted:
- name: docker cache
uses: actions/cache@v3
if: steps.changes.outputs.stack == 'true'
continue-on-error: true
with:
path: .docker-cache
key: ${{ runner.os }}-docker
restore-keys: |
${{ runner.os }}-docker
@madhavajay so, can you try adjusting your filter for me?
- name: Cache OpenmineD Container Image
uses: Broadshield/action-docker-layer-caching@main
with:
filter: reference="*/openmined*:*"
key: docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
restore-keys: |
docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}
docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}
docker-layer-caching-${{ github.workflow }}
docker-layer-caching-
@jamie-wearsafe im still seeing no cache. Could this be because of dockerx or something?
https://github.com/OpenMined/PySyft/runs/7245735124?check_suite_focus=true
@madhavajay there will be no cache until the run completes successfully the first time. Is this a consecutive run?
@jamie-wearsafe shouldnt that happen in the final post step:
Post job cleanup.
/usr/bin/docker image ls --format={{.ID}} {{.Repository}}:{{.Tag}} --filter=dangling=false --filter=reference="*/openmined*:*"
There is no image to save.
