Better observability / error logging

[chanwit]

[tomhuang12]

I am doing some research on how this is implemented in kustomize-controller. Looks like it's using github.com/fluxcd/pkg/runtime/metrics which is already used in various places to record readiness and suspension. The only thing we are missing is recording duration like this.

I think I am missing a big piece here on what you guys are expecting for this issue. Any examples would be great! Thanks!

[phoban01]

I think this is largely related to code structure. The kustomize controller has very clear reconcile phases and the logging, metrics and events are consistent as a result. I think I might make sense to merge the multi-tenancy change first and then tackle this work, otherwise the sands could be shifting beneath our feet a little bit.