Add processing times to OpenTelemetry metrics

[timbussmann]

Describe the suggested improvement

The NServiceBus OpenTelemetry integration currently promotes the following metrics:

• nservicebus.messaging.successes: Total number of messages processed successfully by the endpoint
• nservicebus.messaging.fetches: Total number of messages fetched from the queue by the endpoint
• nservicebus.messaging.failures: Total number of messages processed unsuccessfully by the endpoint

The NServiceBus.Metrics package supports more metrics though, for example

• Critical Time
• Processing Time
• Queue Length

These metrics are currently not available when using the OpenTelemetry integration instead of the Metrics package. When using OpenTelemetry, it does only add noise and overhead to also rely on the NServiceBus.Metrics package for these additional metrics.

Describe the suggested solution

Add additional metrics to the built-in OpenTelemetry metrics, e.g. Critical time and processing time.

Additional Context

No response

[cjoergensen]

Does the NServiceBus.Metrics package expose the metrics via OpenTelemetry by default, or does it require custom code?

[mauroservienti]

@cjoergensen, the Metrics package was designed before the advent of OTel. It requires custom code to capture and report those metrics to an OpenTelemetry collector. I'm not sure the data exposed by the Metrics package have all the needed information to plugin into the OTel activities model.

More details in our documentation.

[bbrandt]

FYI, stay tuned! Great things are on the way according to David Boike!

https://github.com/Particular/docs.particular.net/issues/6446#issuecomment-1934468629

[bbrandt]

Workaround to get these metrics in App Insights without the need for NServiceBus.Metrics: https://docs.particular.net/samples/open-telemetry/application-insights/

And Prometheus: https://docs.particular.net/samples/open-telemetry/prometheus-grafana/

[SzymonPobiega]

The missing metrics have been implemented in https://github.com/Particular/NServiceBus/pull/7097 and https://github.com/Particular/NServiceBus/pull/7095. They will be released soon as part of 9.1