Docs don't mention caching at all

[mvdan]

They used to - noticed some commits and previous issues - but the current docs page doesn't mention them at all: http://plugins.drone.io/drone-plugins/drone-docker/

What is the recommended way to handle this? Even if the docs just covered the basic case of "I don't want to re-download my FROM image every single time", that would be a great step forward.

[mvdan]

One way that would be interesting for it to work is if the plugin checked if the host's Docker already had that image. That is, using the host's image cache as a read-only cache.

That way, if I'm building my deployment images on an Alpine base, all I would need to do for them to be faster is make sure that the base image is cached in the Drone Agent host.

[tboerger]

Today you can use some drone cache plugins for that if you change the docker storage path, but don't use the s3-cache plugin, looks like this won't work for it now.

[mvdan]

Making it work with the regular caching plugins seems like a good idea. An example in the docs would still be great, though.

[Sharsie]

We really tried hard to implement caching using a plugin, maybe you can correct me if we went wrong about it, but here's what we tried and failed at each and every one of the attempts (resolution at the end):

1. Backup/Restore of docker/image and docker/{aufs/overlay2} In this case the cache worked, however with each new build, we kept getting more and more layers and diffs which were not deleted after docker system prune -f After a couple of hundred builds we ended up with cache of several GB for each one of the projects (php apache2 based images with additional 200mb of changing data) This actually resulted in a major slowdown of builds where the cache restore/rebuild steps would take a few times longer than if the cache was not used at all

2. Using a docker service This is fine and dandy until the point where we could not start the docker service and plugins/docker with the same --data-root/--graph, which for obvious reasons did not work. The idea was to use docker service for save/loading of cached image so the layers are available for plugins/docker step. If different --data-root/--graph was used then we could not share the layers between the docker service and plugins/docker. So this led us to drropping the plugins/docker image for publishing and using own docker image with customized build steps and --cache-from parameter. This actually worked, but it was such a dirty solution that we did not really want to go that way and started thinking about PR for plugins/docker to have this implemented correctly.

And this is where we found out there is a PR #151 from september which effectively solves the problem. However, there are still a couple of things that might need further discussion before it is finally merged.

[depado]

I'm also interested in this as the plugin keeps pulling the whole images every time, which is honestly kind of long for multi-stage builds that uses 2 or 3 different images.

I have no idea how to use the caching mechanism for this unfortunately and can't seem to find a proper solution in the issues.

[arno01]

I would like to share with my steps which allowed me to use the docker image caching in Drone CI without the need of using trusted nor privileged git repository options.

To enable docker image caching all you need is:

• DRONE_VOLUME=/tmp/drone-cache:/cache drone-server variable;
• storage_path: /drone/docker and use_cache: true parameters for plugins/docker job;
• drillster/drone-volume-cache of course ;-)

docker-compose.yml

version: '3'

networks:
  oasis: {}

services:
  drone-server:
    restart: unless-stopped
    image: drone/drone:0.8
    networks:
      - oasis
    volumes:
      - /srv/data/drone/lib/drone:/var/lib/drone/
    environment:
      - DRONE_OPEN=false
      - DRONE_ADMIN=REDACTED
      - DRONE_HOST=https://REDACTED
      - DRONE_SECRET=REDACTED
      - DRONE_GITEA=true
      - DRONE_GITEA_URL=https://REDACTED
      - DRONE_GITEA_PRIVATE_MODE=true
      - DRONE_VOLUME=/tmp/drone-cache:/cache

  drone-agent:
    restart: unless-stopped
    image: drone/agent:0.8
    networks:
      - oasis
    command: agent
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=drone-server:9000
      - DRONE_SECRET=REDACTED

.drone.yml

pipeline:
  restore_cache:
    image: drillster/drone-volume-cache:latest
    restore: true
    mount:
      - /drone/docker
    # volumes:
    #   - /tmp/drone-cache:/cache

  publish:
    image: plugins/docker:17.12
    repo: andrey01/REDACTED
    tags:
      - latest
      - ${DRONE_COMMIT_SHA}
    secrets:
      - docker_username
      - docker_password
    storage_path: /drone/docker
    use_cache: true
    when:
      event: [push, tag]
      branch: master

  rebuild_cache:
    image: drillster/drone-volume-cache:latest
    rebuild: true
    mount:
      - /drone/docker
    # volumes:
    #   - /tmp/drone-cache:/cache

After this is set, Drone will keep the docker image cache under: /tmp/drone-cache/<username>/<gitrepo>/1/drone/docker/

When the restore_cache job is running, you can spot it is running rsync from /cache/<username>/<gitrepo>/1//drone/docker/ to /drone/docker inside the drone pipeline. And the /cache is mounted from host:/tmp/drone-cache across the whole pipeline as DRONE_VOLUME drone-server variable was leveraged.

Snap:

# ps auxwwf
...
root       896  0.4  0.0   6216  1532 ?        Ss   21:01   0:00  |       \_ /bin/bash /usr/local/cacher.sh
root       964 30.8  0.1  38192  2320 ?        R    21:01   0:05  |           \_ rsync -aHA --delete /cache/<username>/<gitrepo>/1//drone/docker/ /drone/docker
root       965  4.7  0.0  31720  1816 ?        S    21:01   0:00  |               \_ rsync -aHA --delete /cache/<username>/<gitrepo>/1//drone/docker/ /drone/docker
root       966 45.6  0.1  38116  2356 ?        S    21:01   0:08  |                   \_ rsync -aHA --delete /cache/<username>/<gitrepo>/1//drone/docker/ /drone/docker
...

The docker image caching helped to reduce the build time for one of my repositories from 4m39s to 51s.

References

• http://plugins.drone.io/drillster/drone-volume-cache/
• http://docs.drone.io/manage-registry-credentials/#image-caching-behavior

Missing documentation

• This documentation does not have storage_path variable documented for some reason. Edit PR to update it
• DRONE_VOLUME is not reflected in the documentation for some reason.. I have found information about it only here and in the code Edit PR to update the doc

FYI: @bradrydzewski

[Sharsie]

@arno01 We've had similiar caching mechanism previously, the trouble was, that there were some zombie layers getting cached with each build. See my previous comment and check whether it also does not affect you.

I think we debugged it with a dockerfile that had 2 steps... first step was cachable, something like COPY on a file that has not changed... the other step was never cached, i.e. generating new file with each build

After every build has finished, we've found out there are 2 more layers added to the filesystem (and the ones from the previous build were never purged)

[kav]

Worth calling out the OP comment; while caching of build layers would be cool and whatnot a great first step would be caching of external images. Might be worth tilting at that and avoiding the how to cache build layers questions and associated issues for the moment.

[laszlocph]

Mounting the layer cache into the plugin is not recommended I think.

Instead, you can use cache-from that is portable even to drone cloud.

kind: pipeline
name: default

steps:
  - name: docker-builder
    image: plugins/docker
    settings:
      repo: laszlocloud/cache-from-test
      tags: latest
      cache_from: "laszlocloud/cache-from-test:latest"
      dry_run: true

More on this here: https://laszlo.cloud/how-using-cache-from-can-speed-up-your-docker-builds-in-droneci

[marksteve]

Got around this by mounting the docker storage path of the plugin to the host:

kind: pipeline
name: default

steps:

- name: build
  image: plugins/gcr
  volumes:
  - name: docker-gcr
    path: /var/lib/docker

volumes:
- name: docker-gcr
  host:
    path: /var/lib/docker-gcr

One huge caveat tho: You only run one pipeline/step at a time :/

[fungiboletus]

cache_from is not helpfull with multi-stage builds. Why DroneCI doesn't use a cache by default ?

[bradrydzewski]

Why DroneCI doesn't use a cache by default ?

For security reasons. If we used the host machine cache, someone could send a malicious pull request that overwrites commonly used images (e.g. golang, or node) and replace with malicious versions of images. These could be used to capture secrets, source code and more.

Also to prevent race condition where two builds are running at the same time on the same machine, and both trying to create an image with the same tag name (e.g. :latest), which would be problematic.

[fungiboletus]

Oh, I never thought about that. It would be important to mention it in the documentation if it's updated for caching images.

[thomasf]

I believe that docker build support multiple --cache_from entries so that option should maybe be turned into a list if it already isn't. The big issue for me is though that all layers in a typical larger docker build are in intermediate steps and we cannot push those easily to a repository now given how drone-docker works.

In the example below the most important steps to be able to cache are the npm ci and pip wheel but they are not really made available with drone-docker right now.

from node:10-stretch as jsbuilder
run yarn global add [email protected] [email protected] jest prettier
copy jsapps/package*.json /opt/barfoo/jsapps/
workdir /opt/barfoo/jsapps
run npm ci
copy jsapps/ /opt/barfoo/jsapps/
run npm run prod && rm -rf node_modules

from python:3.7 as pythonbuilder
run apt-get update -qq && \
    apt-get install -y gettext && \
    rm -rf /var/lib/apt/lists/*
workdir /opt/barfoo
copy requirements* /opt/barfoo/
copy requirements /opt/barfoo/requirements
arg PIP_FIND_LINKS=/wheel
run pip -q \
        wheel \
        --wheel-dir /wheel \
        --find-links $PIP_FIND_LINKS \
        --no-cache-dir \
        -r requirements.txt \
        -r requirements/requirements-docker.txt
run pip install \
        --find-links /wheel \
        --no-index \
        --no-cache-dir \
        -r requirements.txt \
        -r requirements/requirements-docker.txt
# ...

from python:3.7
workdir /opt/barfoo
# ...
copy --from=pythonbuilder /wheel /wheel
copy requirements* /opt/barfoo/
copy requirements /opt/barfoo/requirements
run pip install \
        --find-links /wheel \
        --no-index \
        --no-cache-dir \
        -r requirements.txt \
        -r requirements/requirements-docker.txt
copy . /opt/barfoo
copy --from=pythonbuilder /opt/barfoo /opt/barfoo
copy --from=pythonbuilder /opt/barfoo_static /opt/barfoo_static
# ...

I want the Dockerfile to be self contained so I don't want separate drone steps to do stuff required to build the full image because I want it to build anywhere only using docker build. I do the drone stuff with volume mounted caces for npm/pip/... as well for the linting steps of the build but for image publishing I want it to build from one easy to understand source.

[bradrydzewski]

cache_from [...] should maybe be turned into a list if it already isn't

Just confirming that cache_from is already a list [1] [1] https://github.com/drone-plugins/drone-docker/blob/master/docker.go#L50

[masterkain]

in my (hairy) quest for trying to enable proper docker image layer caching in a k8s env without agents with drone may I steal the thread and ask: how to pass plugins/docker settings to a plugins/ecr step? I see in code that in the end ecr does call the docker plugin, any way to do that? trying the cache-from method but I need ecr step.

[tboerger]

All docker options should also directly be available to the ecr plugin. Settings are injected as env variables. You have written cache-from, but it got to be cache_from.

[thomasf]

I would argue that the absolute worst feature of the official (or whatever you'd like to call them) drone plugins is that the don't warn when being supplied with invalid argument names. This issue comes up so often...

Preferably all plugins should also have a PLUGIN_PLUGIN_DEBUG option to turn on verbose logging because that's the second thing that I find usually goes wrong, plugins can have fairly complicated inner workings and just a simple error message might not always be enough when figuring out whats going wrong.

I don't think that I can count the times that I personally have had to add logging to a plugin to understand why it fails to do what it's supposed to on two hands.

While these two issues range for mildly to very annoying for me personally I would just have to guess how annoying it is for people who don't know how to modify the plugins to inspect what it is doing and it's probably a lot more problematic and frustrating experience than for me...

[tboerger]

There are plans to generate plugins from a better standard lib which includes logging and flag definition. But that requires further discussion.

[lehno]

Im using cache_from and ECR, for small docker images, it works, for my bigger ones, it builds everything from scratch. Any tips?

[lehno]

Its a multi-stage dockerfile, so it fails because the first build is not available =/

[ywanbing]

so? Is there any good way to support the mirror cache now? It is really too slow to pull the build image again every time.

It's been several years. Is there any good way?