Meshctl check command - Improve diagnostic tools

[nimrodoron]

Improve diagnostic tools - the meshctl check command shouldn't show green across the board when the setup effectively doesn't work.

1. Don't show green if an unsupported version of Istio has been identified remotely and can be expected not to work.
2. Don't show green if the remote Istiod hasn't yet been restarted after being added to a VirtualMesh.
3. Don't show green if workloads haven't been restarted and won't work.
4. Don't show green if Istio is not configured correctly as per the Gloo Mesh requirements and certain features can be expected not to work.

[linsun]

Hi @nimrodoron great suggestions!

For 2, Istio recently (1.12) improves on auto detect intermediate certs change without restart of istiod, so restart should not be required in newer Istio. See https://istio.io/latest/news/releases/1.12.x/announcing-1.12/change-notes/#security, the 2nd bullet: Added support to istiod to notice cacerts file changes via the AUTO_RELOAD_PLUGIN_CERTS environment variable. (Issue #31522). GM can be configured to leverage this soon.

For 3, Could you explain why you expect workloads to be restarted?

[liorokman]

For 3, it's not about automatically restarting workloads. It's about being able to run a diagnostic that will detect problematic situations.

For example, once Istio is newly connected to a VirtualMesh and restarted then the intermediate CAs change. Any workload that was already running when this happened will simply lose its connection to Istio, and be stuck in limbo. The workloads themselves need to be restarted so that the Istio provided sidecar can continue its conversation with the local IstioD.

It would be useful for the meshctl check command to be able to detect that some workloads are currently not working correctly and need to be restarted.