enhancement: apply `EnvoyGateway` without manually restarting the EG deployment

[zhaohuabing]

Description: Currently, the EG deployment must be manually restarted after updating the ConfigMap containing EnvoyGageway configuration.

To streamline the process, we could implement a mechanism where EG automatically detects the CM changes and restart itself.

To achieve this, EG can either:

• Use a timestamp annotation in EG deployment template to trigger a restart.
• Reload the config and restart all runners without restarting itself.

[ShyunnY]

We can implement a watcher through informer to help us hot refresh the controller configuration.

This looks like a great feature and I'm looking forward to it.

[shawnh2]

+1 for this one

xref: #864

[zirain]

Will filewatcher help this? I recall there's same requirement in istio for istio.istio-system configmap.

[zirain]

On the otherhand, a filewatch require a long time to catch a event, informer is a better choice?

[arkodg]

+1 to a non k8s specific way to solve this e.g. using https://github.com/fsnotify/fsnotify

[zirain]

I recall there's some issue when using fsnotiry in kubernetes, so istio use a wrapper https://github.com/istio/pkg/blob/master/filewatcher/filewatcher.go

[ShyunnY]

I think that in most scenarios where the gateway configuration needs to be refreshed, we may pay more attention to real-time performance. Can we accept the delay caused by fsnotiry?

[shawnh2]

IMO, for Kubernetes provider, the Informer would be much accurate and fast than watching fsnotify event. According to the ConfigMaps update strategy, the potential delay of automatically update mounted ConfigMaps (which basically imply using the fsnotify approach) lies in kubelet sync period and cache propagation.

The fsnotify could also work, but this approach may be very suitable for the File provider.

• If we seeking code generality, the fsnotify approach works for both.
• If we seeking the accuracy and realtime, the Informer for k8s and fsnotify for file provider are suggested.

[arkodg]

if the fsnotify approach enables the changes under 5s, its good enough, imo

[zirain]

if the fsnotify approach enables the changes under 5s, its good enough, imo

I recall it may take 30s in the worst.