Container changes in`Deployment` => "element [0] missing"

[Dominik-K]

Using this deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
              name: http
              protocol: TCP

If I change a field in the containers part, e.g. the containerPort, kubediff returns:

## test/nginx (Deployment)

.spec.template.spec.containers: 'element [0]' missing

[tomwilkie]

Yeah, thats an unfortunate side effect of https://github.com/weaveworks/kubediff/pull/31.

@awh can we improve this?

[awh]

We can, by teaching kubediff to sort certain lists e.g. .spec.template.spec.containers should be sorted by .name before comparison.

[ademariag]

is there an update on this? or a workaround?

[carlosdagos]

Getting the same issue. An update would be awesome. Thanks for kubediff!

[damienleger]

I've coded a workaround for this. It's far from perfect because tests are broken and all but that fits my needs. Looks interesting to share anyway.

My understanding is that after the change https://github.com/weaveworks/kubediff/pull/31 every change in any deep after a list results in 'element[X] missing' issue we are talking about. Hence if you change the image of a container (the most likely on a daily change), you have this issue, making the tool quite useless imho. The change make sense in a way that if you remove or add element in a list from running or wished configuration, differences between elements get mixed up and you end up with an horrible diff output e.g.: you remove wished var1 then running var1 is diffed with wished var2, running var2 is diffed with wished var3 and so on... Luckily changes not happening much except for environment variable list.

I then did the following change. Reverted the old behavior for list having same size, and if not return 'element missing' except for .spec.template.spec.containers[X].env whish have a special treat.

Let's say you running on this

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.1
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
          env:
            - name: name1
              value: value1
            - name: name2
              value: value2
            - name: name3
              value: value3

and want this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
          env:
            - name: name1
              value: value1.1
            - name: name4
              value: value4

With my hack, kubediff output is:

## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].image: 'nginx:1.14.2' != 'nginx:1.14.1'
## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].env: Unequal lengths: 2 != 3
## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].env: Diff: name4 in wished config only
## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].env: Diff: name2 in running config only
## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].env: Diff: name3 in running config only
## test/nginx (Deployment.v1beta1.extensions)

.spec.template.spec.containers[0].env[0].value: 'value1.1' != 'value1'

Patch is attached. Edit kubedifflib/_diff.py and build container with build instructions in doc except "make .uptodate" instead of "make" to bypass tests. patch.txt

[collinbachi]

Any update here? I have to imagine comparing changes to containers is pretty crucial to most people's use cases.

[chancez]

Trying kubediff for the first time and I get this on an array inside one of the values for my HelmRelease.

Eg:

kubediff consul grafana loki prometheus-k8s vault
## loki/loki (HelmRelease.v1beta1.flux.weave.works)

.spec.values.loki.config.schema_config.configs: 'element [0]' missing

With the following HelmRelease:

apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  name: loki
  namespace: loki
  annotations:
    fluxcd.io/automated: "false"
spec:
  releaseName: loki
  chart:
    repository: https://grafana.github.io/loki/charts
    name: loki-stack
    version: 0.23.0
  values:
    loki:
      enabled: true

      service:
        type: NodePort
        port: 3100
        nodePort: 32004

      serviceMonitor:
        enabled: true

      replicas: 3

      env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: my-aws-secret
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: my-aws-secret
              key: AWS_SECRET_ACCESS_KEY
        # Consul runs the agent as a daemonset on a host port, so use the downward
        # API to get the hostIP
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP

      extraArgs:
        # Use extraArgs to set the -consul.hostname configuration using an
        # environment variable from the downward API.
        "consul.hostname": "$(HOST_IP):8500"

      config:
        auth_enabled: false
        server:
          http_listen_port: 3100
          http_server_idle_timeout: "30s"

          graceful_shutdown_timeout: "30s"
          # 64MB = 67108864 bytes
          grpc_server_max_send_msg_size: 67108864

        limits_config:
          enforce_metric_name: false
          reject_old_samples: true
          reject_old_samples_max_age: "168h"

        ingester:
          chunk_idle_period: 15m
          # 256KB
          chunk_block_size: 262144

          # max attempts to transfer in-memory data to another ingester replica
          # before deciding to flush to storage.
          # max_transfer_retries: 60

          lifecycler:
            ring:
              replication_factor: 3
              heartbeart_timeout: 1m
              kvstore:
                store: consul
                prefix: "loki-collectors/"
                consul:
                  # "host" is unset in this configuration file because it's set
                  # using extraArgs due to the value coming from an environment
                  # variable.
                  httpclienttimeout: "20s"
                  consistentreads: true

            # num_tokens: 512
            join_after: "30s"

        ingester_client:
          grpc_client_config:
            # 64MB = 67108864 bytes
            max_recv_msg_size: 67108864

        storage_config:
          aws:
            s3: s3://us-west-1/czibolski-rigetti-test
            dynamodbconfig:
              dynamodb: dynamodb://us-west-1

        schema_config:
          configs:
            - from: 2019-12-01
              store: aws
              object_store: s3
              schema: v9
              index:
                prefix: czibolski_loki_test_index_
                period: 168h

        chunk_store_config:
          max_look_back_period: "30s"

        table_manager:
          retention_deletes_enabled: true
          retention_period: 336h

    promtail:
      enabled: true

Here's the resource in the cluster

k get hr -n loki loki -o yaml

apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  annotations:
    fluxcd.io/automated: "false"
    fluxcd.io/sync-checksum: 34e416b2146d9562dba3c0aab372f593c96a2515
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"flux.weave.works/v1beta1","kind":"HelmRelease","metadata":{"annotations":{"fluxcd.io/automated":"false","fluxcd.io/sync-checksum":"34e416b2146d9562dba3c0aab372f593c96a2515"},"labels":{"fluxc
  creationTimestamp: "2019-12-13T17:44:36Z"
  generation: 1
  labels:
    fluxcd.io/sync-gc-mark: sha256.YI3vseFaMevalw7yAnP6WIQ8a5jp58PUacjQdk47ZRQ
  name: loki
  namespace: loki
  resourceVersion: "49026"
  selfLink: /apis/flux.weave.works/v1beta1/namespaces/loki/helmreleases/loki
  uid: a6c02724-4b6d-4ca5-a306-f8cdffc96b18
spec:
  chart:
    name: loki-stack
    repository: https://grafana.github.io/loki/charts
    version: 0.23.0
  releaseName: loki
  values:
    loki:
      config:
  chart:
    name: loki-stack
    repository: https://grafana.github.io/loki/charts
    version: 0.23.0
  releaseName: loki
  values:
    loki:
      config:
        auth_enabled: false
        chunk_store_config:
          max_look_back_period: 30s
        ingester:
          chunk_block_size: 262144
          chunk_idle_period: 15m
          lifecycler:
            join_after: 30s
            ring:
              heartbeart_timeout: 1m
              kvstore:
                consul:
                  consistentreads: true
                  httpclienttimeout: 20s
                prefix: loki-collectors/
                store: consul
              replication_factor: 3
        ingester_client:
          grpc_client_config:
            max_recv_msg_size: 67108864
        limits_config:
          enforce_metric_name: false
          reject_old_samples: true
          reject_old_samples_max_age: 168h
        schema_config:
          configs:
          - from: "2019-12-01"
            index:
              period: 168h
              prefix: czibolski_loki_test_index_
            object_store: s3
            schema: v9
            store: aws
        server:
          graceful_shutdown_timeout: 30s
          grpc_server_max_send_msg_size: 67108864
          http_listen_port: 3100
          http_server_idle_timeout: 30s
        storage_config:
          aws:
            dynamodbconfig:
              dynamodb: dynamodb://us-west-1

[Smana]

The patch from @damienleger works great. thanks

[gouravsw]

@damienleger I have downloaded the kubediff repo and using the ./kubediff I am trying to find the differences in k8s state and manifest files. I have modified the _diff.py file by removing the lines and adding new lines.

Even after that I am still getting the similar issues.

As per your comment "build container with build instructions in doc except "make .uptodate" instead of "make" to bypass tests."

Could you please share the link of the document which you are referring?

[damienleger]

hello @gouravsw, tbh I've did that tweak for my previous job so I'm not working on this anymore. That being said, when referring to "doc" I meant this --> https://github.com/weaveworks/kubediff#build

So by replacing make by make .uptodate, here it is:

mkdir -p $GOPATH/src/github.com/prometheus && cd "$_"
git clone [email protected]:prometheus/client_golang.git
mkdir -p $GOPATH/src/github.com/weaveworks && cd "$_"
git clone [email protected]:weaveworks/kubediff.git
cd kubediff
make .uptodate