Expose concurrency based metrics

[dhumphries-sainsburys]

In a separate issue we are looking at issues with concurrency but it is hard to get a true grasp of any problem real or perceived without concrete metrics. We have some for things interacting with flux but the actual inner workings of things such as the kustomize-controller are sadly hidden from us so we are left guessing at what is happening with. Is it possible to expose some useful concurrency metrics to prometheus such as:

queue depth thread timings active thread counts

Whilst the above are things that we think could be useful for our current state more metrics wouldn't be a bad thing (even better if we could also turn off any that aren't needed)