Impossible to redeploy a non blue-green application

[lunika]

Bug Report

Expected behavior/code

When I deploy an application with blue-green settings set to false and parameters changed in a template (DeploymentConfig mostly), I expect a to deploy without error with a rolling-update strategy

Actual Behavior

The deploy fail with this kind of error:

TASK [OpenShift deployments with deployment_stamp[d-190916-09h04m48s] must be present] *******************************************************************************************************
failed: [local] (item=apps/learninglocker/templates/services/app/dc_api.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_api.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-api\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-api\\\\\", \\\\\"service\\\\\":\\\\\"api\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-api\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"version\\\\\":\\\\\"v2.8.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-api\\\\\", \\\\\"service\\\\\":\\\\\"api\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/app/dc_ui.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_ui.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-ui\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-ui\\\\\", \\\\\"service\\\\\":\\\\\"ui\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-ui\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-ui\\\\\", \\\\\"service\\\\\":\\\\\"ui\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/app/dc_worker.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_worker.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-worker\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-worker\\\\\", \\\\\"service\\\\\":\\\\\"worker\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-worker\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"service\\\\\":\\\\\"worker\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-worker\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/xapi/dc.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/xapi/dc.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-xapi\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-xapi\\\\\", \\\\\"service\\\\\":\\\\\"xapi\\\\\", \\\\\"version\\\\\":\\\\\"v2.9.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-xapi\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"version\\\\\":\\\\\"v2.9.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-xapi\\\\\", \\\\\"service\\\\\":\\\\\"xapi\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

Steps to Reproduce

1. Deploy a first version of a non blue-green compatible application (like learning locker)
2. Change the image tag version
3. Deploy it again

Environment

• Arnold version: 2.7.0 and master
• Platform: our containers.

Possible Solution

I tried to use the force parameter in the step OpenShift deployments with deployment_stamp[{{ deployment_stamp }}] must be {{ deployment_state | default('present') }} in manage_app.yml task but I have a new error I don't really understand:

failed: [local] (item=apps/learninglocker/templates/services/app/dc_api.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_api.yml.j2", "msg": "Failed to replace object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"deploymentconfigs \\\\\"learninglocker-api\\\\\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-api\",\"kind\":\"deploymentconfigs\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: 0x0: must be specified for an update\",\"field\":\"metadata.resourceVersion\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

[madmatah]

I've just got the same issue with a non blue/green application.

My DeploymentConfig definition was something like this :

apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: elasticsearch
    service: elasticsearch
    version: "{{ elasticsearch_image_tag }}"
    type: es-node
  name: "elasticsearch-node"
  namespace: "{{ project_name }}"
spec:
  replicas: {{ elasticsearch_nodes }}
  template:
    metadata:
      labels:
        app: elasticsearch
        service: elasticsearch
        version: "{{ elasticsearch_image_tag }}"
        type: es-node
    spec:
      containers:
        - image: {{ elasticsearch_image_name }}:{{ elasticsearch_image_tag }}
          name: elasticsearch

It seems that once a DeploymentConfig resource is deployed, it cannot be patched if the labels defined in spec.template.metadata.labels change. They are used as a label selector to target the resource to patch.

I removed the dynamic parts this section and it worked ! :tada:

apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: elasticsearch
    service: elasticsearch
    version: "{{ elasticsearch_image_tag }}"
    type: es-node
  name: "elasticsearch-node"
  namespace: "{{ project_name }}"
spec:
  replicas: {{ elasticsearch_nodes }}
  template:
    metadata:
      labels:
        app: elasticsearch
        service: elasticsearch
        type: es-node
    spec:
      containers:
        - image: {{ elasticsearch_image_name }}:{{ elasticsearch_image_tag }}
          name: elasticsearch

Once deployed and patched :

$> oc get dc --show-labels=true
NAME                 REVISION   DESIRED   CURRENT   TRIGGERED BY   LABELS
elasticsearch-node   2          3         3         config         app=elasticsearch,service=elasticsearch,type=es-node,version=6.3.1

[jmaupetit]

What if we use the force: true flag during DC creation/patching?

[madmatah]

What if we use the force: true flag during DC creation/patching?

It doesn't work (see @lunika's comment in Possible solution) I also tried merge_type: merge, merge_type: strategic-merge and apply: yes.

[jmaupetit]

So, in case of version upgrade, should we delete and then create the object instead of patching it?

[madmatah]

If we do it that way (delete / create), all interesting features of the DeploymentConfig (like rollback or deployment strategies) will not be usable.

What do you think about the solution i proposed ? Dynamic labels are still usable for us and re-deployment is working.

[jmaupetit]

If thought it was a test to understand the issue, not a solution per se :sweat_smile:

So if we follow your recommendation, we will loose version labels in pods, right? Knowing that the information is in the image tag, I think it's a reasonable approach.

[madmatah]

You're right, we lose the version label in pods but we keep it in deploymentconfig.