[Feature Request] Customize metric bucket sizes

[lorensr]

Is your feature request related to a problem? Please describe.

From a user:

We noticed the metric caps out at 10 seconds. If we start a large batch workflows we are generally fine waiting for them to process for 10-15 minutes. Given the 10 second maximum it’s pretty hard to tell how busy the workflow queue is.

(small number of workflow task slots)

[cretz]

I have changed the title to reflect the general desire to customize bucket sizes. Unsure if this can be done on a per metric basis or just in general for all durations/histograms.

[punit-kulal]

Also, if we are to record our own metrics, even those use the default buckets, which are not always ideal