skaffold icon indicating copy to clipboard operation
skaffold copied to clipboard
verified publisher icon GoogleContainerTools

switching from docker to kaniko in different profiles is hard to configure

Open balopat opened this issue 5 years ago • 6 comments

Today, if you want to switch all your builder types to kaniko from local docker, you'll have to repeat all your artifacts. This is cumbersome so people resort to using YamlAnchors. E.g. https://github.com/ForgeRock/forgeops/blob/master/skaffold.yaml

The hydrated version of this yaml is this:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
  - image: am
    context: docker/7.0/am
    docker:
      dockerfile: Dockerfile
  - image: amster
    context: docker/7.0/amster
    docker:
      dockerfile: Dockerfile
  - image: idm
    context: docker/7.0/idm
    docker:
      dockerfile: Dockerfile
  - image: ds-cts
    context: docker/7.0/ds
    docker:
      dockerfile: cts/Dockerfile
  - image: ds-idrepo
    context: docker/7.0/ds/
    docker:
      dockerfile: idrepo/Dockerfile
  - image: forgeops-secrets
    context: docker/forgeops-secrets
    docker:
      dockerfile: Dockerfile
  - image: ig
    context: docker/7.0/ig
    docker:
      dockerfile: Dockerfile
  tagPolicy:
    sha256: {}
  local:
    concurrency: 1
deploy:
  kustomize:
    paths:
    - ./kustomize/overlay/7.0/all
  statusCheckDeadlineSeconds: 600
profiles:
- name: forgeops
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/forgeops
- name: nightly
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: amster
      context: docker/7.0/amster
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: idm
      context: docker/7.0/idm
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-cts
      context: docker/7.0/ds
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: cts/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-idrepo
      context: docker/7.0/ds/
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: idrepo/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: forgeops-secrets
      context: docker/forgeops-secrets
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    tagPolicy:
      sha256: {}
    cluster:
      pullSecretName: kaniko-secret
      namespace: kaniko
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/nightly
- name: smoke
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: amster
      context: docker/7.0/amster
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: idm
      context: docker/7.0/idm
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-cts
      context: docker/7.0/ds
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: cts/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-idrepo
      context: docker/7.0/ds/
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: idrepo/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: forgeops-secrets
      context: docker/forgeops-secrets
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    tagPolicy:
      gitCommit: {}
    cluster:
      pullSecretName: kaniko-secret
      namespace: kaniko
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/smoke
- name: default-kaniko
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: amster
      context: docker/7.0/amster
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: idm
      context: docker/7.0/idm
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-cts
      context: docker/7.0/ds
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: cts/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-idrepo
      context: docker/7.0/ds/
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: idrepo/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: forgeops-secrets
      context: docker/forgeops-secrets
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    tagPolicy:
      sha256: {}
    cluster:
      pullSecretName: kaniko-secret
      namespace: kaniko
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/all
    statusCheckDeadlineSeconds: 600
- name: kdev
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: amster
      context: docker/7.0/amster
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: idm
      context: docker/7.0/idm
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-cts
      context: docker/7.0/ds
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: cts/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: ds-idrepo
      context: docker/7.0/ds/
      kaniko:
        flags:
        - --single-snapshot
        dockerfile: idrepo/Dockerfile
        image: gcr.io/kaniko-project/executor:v0.20.0
    - image: forgeops-secrets
      context: docker/forgeops-secrets
      kaniko:
        flags:
        - --single-snapshot
        image: gcr.io/kaniko-project/executor:v0.20.0
    tagPolicy:
      sha256: {}
    cluster:
      pullSecretName: kaniko-secret
      namespace: kaniko
  deploy:
    kustomize:
      paths:
      - ./dev
- name: security
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/security
- name: small
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/small
- name: medium
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/medium
- name: large
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/large
- name: platform-deployment
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: idm
      context: docker/7.0/idm
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/lodestar
- name: ig-only
  build:
    artifacts:
    - image: ig
      context: docker/7.0/ig
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/ig-only
- name: am-only
  build:
    artifacts:
    - image: am
      context: docker/7.0/am
    - image: amster
      context: docker/7.0/amster
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    tagPolicy:
      gitCommit: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/am-only
- name: idm-only
  build:
    artifacts:
    - image: idm
      context: docker/7.0/idm
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/idm-only
- name: ds-only
  build:
    artifacts:
    - image: ds-cts
      context: docker/7.0/ds
      docker:
        dockerfile: cts/Dockerfile
    - image: ds-idrepo
      context: docker/7.0/ds/
      docker:
        dockerfile: idrepo/Dockerfile
    - image: forgeops-secrets
      context: docker/forgeops-secrets
    tagPolicy:
      sha256: {}
  deploy:
    kustomize:
      paths:
      - ./kustomize/overlay/7.0/ds-only

A smaller example:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
  - image: skaffold-example
  local: {}
deploy:
  kubectl:
    manifests:
      - k8s-*
profiles:
  - name: build-with-kaniko
    build:
      artifacts:
        - image: skaffold-example
          kaniko:
            image: gcr.io/kaniko-project/executor:v0.20.0
            flags:
              - '--single-snapshot'
      cluster: {}

Issues with our current config reflecting in repetition are:

  • kaniko.image, kaniko.flags, etc. are only available on the artifact level. Recommendation: If these would be available on the build.cluster.kaniko level, one wouldn't have to repeat them for each artifact
  • if I specify artifacts.kaniko.image and try to build with local docker, then skaffold fails with:
invalid skaffold config: found a 'kaniko' artifact, which is incompatible with the 'local' builder:

image: skaffold-example
context: .
kaniko:
  flags:
  - --single-snapshot
  dockerfile: Dockerfile
  initImage: busybox
  image: gcr.io/kaniko-project/executor:v0.20.0

To use the 'kaniko' builder, add the 'cluster' stanza to the 'build' section of your configuration. For information, see https://skaffold.dev/docs/pipeline-stages/builders/

Recommendation: skaffold could ignore kaniko related settings on the artifacts when building with docker. This means that artifacts could have at the same time settings for docker, kaniko and GCB (though this last one doesn't exist). In goland this might translate to an artifact being able to become simultaneously a KanikoArtifact and a DockerArtifact - not sure - but our abstraction here is messy.

The final result would be something like this:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
  - image: skaffold-example
    docker: {} # mentioned for explicitness
    kaniko: {} # mentioned for explicitness
  local: {}
deploy:
  kubectl:
    manifests:
      - k8s-*
profiles:
  - name: build-with-kaniko
    build:
      cluster:
        kaniko:
          image: gcr.io/kaniko-project/executor:v0.20.0
          flags:
            - '--single-snapshot'

balopat avatar Jun 19 '20 22:06 balopat

Would something like this make sense:

builders:
- name: default # default is docker, unless otherwise specified
  type: docker
- name: kaniko
   type: kaniko
   options:
       image: gcr.io/kaniko-project/executor:v0.20.0
       flags: ....
   cluster:
       namespace: kaniko 
      pullSecretName: kaniko-secret

Then

build:
  artifacts:
  # uses default builder
  - image:  my-image
    context: docker/my-image 
 - image: my-other-image
    context: docker/my-other-image
   builder: kaniko. # overeride default builder

Or at the profile level

profiles: 
- name: my-cool-profile
  builder:  kaniko # Use kaniko as the default builder for this profile

wstrange avatar Jun 22 '20 14:06 wstrange

What you are suggesting is logically sound and follows the "referencable builder" concept that you kind of introduced using YamlAnchors. However, it introduces a new feature: the ability to specify different builders per artifact.

Do you have a use case for that? Based on the forgerock yaml it seems you either want to build all artifacts with docker or all of them with kaniko. Would the hybrid mode help you?

The hybrid mode can complicate things when you have each of the artifacts potentially built differently (docker, kaniko, gcb) and remove some options for optimizations, but it's far from impossible.

We had a very long discussion we had about per artifact builders where we concluded that we shouldn't support this: https://github.com/GoogleContainerTools/skaffold/issues/1179#issuecomment-433458471

balopat avatar Jun 22 '20 20:06 balopat

We don't really have a use case for a builder per artifact. I got carried away with a bit of bike shedding!

A referencable builder per profile (with a default if omitted) would be quite nice. It would save repeating the builder spec on each profile.

wstrange avatar Jun 22 '20 21:06 wstrange

Is this use case planned to be supported? Thanks

albertocavalcante avatar Jul 26 '21 07:07 albertocavalcante

yes! we are planning to work on this in our proposed build config revamp in Q4

tejal29 avatar Aug 09 '21 18:08 tejal29

Just to get this issue to show some life:

This remains a big unaddressed usability issue with skaffold

clhodapp avatar May 24 '24 23:05 clhodapp