get_params: save does not save image in v5.0.0

[kozak-codes]

I have a multi-stage docker build that I am trying to get caching to work with but when I put the docker resource with get_params: {save: true} it appears that the image is deleted from the folder and is not saved on the next build step.

I am running concourse v5.0.0.

Here is my setup:

resources:

  - name: repo-release
    type: git
    source:
      uri: {{github-repo-uri}}
      branch: release
      private_key: {{github-private-key}}

  - name: frontend-image-base
    type: docker-image
    source: ((redacted))
  - name: frontend-image-builder
    type: docker-image
    source: ((redacted))
  - name: frontend-image
    type: docker-image
    source: ((redacted))

# JOBS
jobs:
  - name: build-frontend
    plan:
      - get: repo
        resource: repo-release
        trigger: true

      - get: frontend-image-builder
        inputs: # use blank inputs to decrease container file sizes
        params:
          save: true
      - get: frontend-image
        inputs:
        params:
          save: true
      - get: frontend-image-base
        inputs:
        params:
          save: true
        #/ End Aggregate


      # START BUILD (placed in do block for success/fail/cancel handles)
      - do:
        # BUILD BUILDER
        - put: frontend-image-builder
           get_params:
             skip_download: true
             save: true
           params:
             target_name: builder
             build: repo
             cache_from:
               - frontend-image-builder
             load_bases:
               - frontend-image-base

        # BUILDER RUNNER
        - put: frontend-image
           get_params:
             skip_download: true
           params:
             build: repo
             target_name: runner
             load_bases:
               - frontend-image-base
               - frontend-builder-new
             cache_from:
               - frontend-builder-new
               - frontend-image
             tag_as_latest: true
        on_success:
        on_failure:
        on_cancel:

When I run fly inspect on put: frontend-image-builder I can see the image in the frontend-image-builder folder.

When I run fly inspect on put: frontend-image I can only see digest, image-id, and tag in the frontend-image-builder folder. It looks like frontend-image-builder is deleting the image and it is not saving over for some reason. Is this a bug or am I doing something wrong here?

My workaround for the moment is to add a get in between the builder and the frontend which seems to be able to pull the latest frontend-image-builder. For example:

...
# START BUILD
      - do:
        # BUILD BUILDER
        - put: frontend-image-builder
          # same as before
          
          # REGET BUILDER BECAUSE CONCOURSE HAS A BUG?
        - get: frontend-builder-new
          resource: frontend-image-builder
          inputs:
          params:
            save: true

        # BUILDER RUNNER
        - put: frontend-image
           # same as before
            load_bases:
              - frontend-image-base
              - frontend-builder-new # note different tag
            cache_from:
              - frontend-builder-new # note different tag
              - frontend-image
...

[vito]

You've got both skip_download and save configured. If skip_download is set there's nothing to download and thus nothing to save. Does it work if you remove skip_download?

[grabekm90]

I have a similar issue with concourse v5.8.0. Please look at my simple example:

  - put: base-image
    params:
      build: test-branch
      dockerfile: test-branch/((base-image-dockerfile-path))
    get_params:
      # with skip_download the image is not saved for prod-image
      # skip_download: true
      save: true
  - put: prod-image
    params:
      load_base: base-image
      build: test-branch
      dockerfile: test-branch/((prod-image-dockerfile-path))
    get_params:
      # now we can skip_download because the image is not used later
      skip_download: true

In this configuration, base-image must be downloaded just after the build. We cannot add skip_download: true for the first image, because the image would not be saved. It is weird because the image was just built :|

[xtremerui]

@grabekm90 are you still having this issue? One thing to quickly check is quote the true for both save and skip_download. Refer to https://github.com/concourse/docker-image-resource/issues/239#issuecomment-448945016.

[grabekm90]

I still have the issue with concourse v7.1.0. Quoting the true did not help. For the record, I cannot set skip_download: true and save: true in just build image - the image is not saved so I cannot use it as a base image and I get an error: no such file or directory. Only when I set skip_download: false, it works.

[grabekm90]

Do you know any solution to how I can cache a just built image? I would like to build an image, push it, and then use it (from cache) as a base image to another image.

[xtremerui]

@grabekm90 seems like from the code here the save won't happen if skip_download is true

https://github.com/concourse/docker-image-resource/blob/master/assets/in#L58-L73

as per https://github.com/concourse/docker-image-resource/issues/263#issuecomment-476709905

So to your question, no you can't use an image from cache after put. It has to be downloaded remotelly. Is there a specific reason that you don't want it to downloaded?

[grabekm90]

It is just a performance issue. I would like to build the next image faster without unnecessary download of just build image and available on the same worker.