[FEATURE] Request for the new methods of workloads restarting: delete and HTTP query

[macrokernel]

The problem Wave may fail to restart workloads in cases when pods or Deployments are controlled by an operator. The operator may recreate the workloads immediately after Wave adds its annotation, which results in the new pods killed and the old pods keep running. This is the case with OpenSearch operator.

The proposed solution - method 1 It would be great if Wave could send HTTP queries specified with annotations. The queries would trigger service restarts at software level. Of course it will only work with the software that is able to restart after receiving an HTTP request. Here is an example of the annotations:

  annotations:
    wave.pusher.com/update-on-config-change: "true"
    wave.pusher.com/restart-method: "http"
    wave.pusher.com/http-method: "GET"
    wave.pusher.com/http-url: "http://my-service.default.svc.cluster.local/action/restart/"

Or a more complex one:

  annotations:
    wave.pusher.com/update-on-config-change: "true"
    wave.pusher.com/restart-method: "http"
    wave.pusher.com/http-method: "PUT"
    wave.pusher.com/http-url: "https://opensearch.opensearch.svc.cluster.local:9200/_plugins/_security/api/ssl/transport/reloadcerts"
    wave.pusher.com/http-data: '{ "foo": "bar" }'
    wave.pusher.com/http-basic-auth-secret: "my-basic-auth-secret"
    wave.pusher.com/http-basic-auth-username-key: "username"
    wave.pusher.com/http-basic-auth-password-key: "password"

The proposed solution - method 2 Simply delete the workload. After the workload is deleted by Wave, the operator would notice this and automatically recreate the workload, resulting in the pods restart. I see this new method as an easy to implement way of restarting workloads controlled by operators, which disallow edition/patching of the resources under their control. This is brute-force, but can be used as the last resort.

Thanks in advance.

[toelke]

Hey.

I understand the problem. Unfortunately, the annotation wave.pusher.com/config-hash contains the hash of the current content of secrets/configmaps and is used by wave to detect if the content changed. So we cannot just remove it.

Does the OpenSearch operator pass annotations on the custom resource to the pods? Or does it have a field for extra pod annotations?

We could then do something like

kind: Pod
metadata:
  annotations:
    wave.pusher.com/update-instead-kind: OpenSearch
    wave.pusher.com/update-instead-name: my-cluster
    wave.pusher.com/update-instead-jsonpath: .spec.configuration.extraPodAttributes

and update the OpenSearch my-cluster instead of the Deployment.

Thoughts?

[macrokernel]

Hi Philipp,

Any modifications to pod template including annotations result in the pod being killed by the operator. However OpenSearch operator allows adding annotations to the deployment metadata:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    foo: bar          <--- can be added manually anytime, can be edited, patched
spec:                                        
  template:                                        
    metadata:                                      
      annotations:  
        bar: baz     <--- can be added only at the OpenSearch Cluster resource creation time, is created by the operator

I am not sure if the same is generally true for all operators, I think it is not. I know some operators that don't use Deployments and directly create Pods.

[jabdoa2]

I have never seen any other operators behave this way. Flux/ArgoCD with Helm Operator can overwrite/restore deployments but it is easy to tell it to exclude certain path as this is a common thing: https://fluxcd.io/flux/components/helm/helmreleases/#ignore-rules. For some reason this is not even needed for Wave.