temporalio
Allow contributions of tags of root worker metricsscope per workflow execution
We are using Temporal to handle workflows for registrations. Those registrations are for a certain region/country. We are sending the metrics to New Relic to create a dashboard to get insights in how many workflows have started, failed, cancelled or completed. This works fine. The region is part of the input that starts the workflow.
However we would like to split those metrics based on the region, we tried several hooks but apart from sending a custom metric on the start of a workflow we aren't able to solve this. What we want is that when a workflow starts, it contributes a tag to the metrics (in our case the region) so that we can use that as an additional dimension/facet in New Relic to group things by.
We are using the Java SDK and haven't found a foolproof way of doing that, apart from a custom metric at the start of the workflow.
What we currently have is we have a single WorkflowServiceStubs which we cannot use to set the region. We tried to create a WorkflowServiceStubs instance per region, but that starts to break as soon as we need to react to signals, we there loose the region and cannot use the proper one.
We tried to create a WorkflowClientInterceptor which doesn't give us access to the correct MetricScope to add tags.
We also tried a ContextPropagator although we can pass the region around there is no way to contribute it as a tag to the metrics.
All in all the only thing that worked was a custom metric at the start of a workflow, but that still leaves out the cancelled, failed and completed events we need to have.
It would be nice to have a way that when starting a workflow, we could add custom tags to the metrics and that those would be added to all events being send to New Relic. Currently it seems to hard or to be to inflexible.
It's not possible right now to set custom tags on core worker metrics from the workflow code. There is no easy, clean and reliable way to achieve and do it with Temporal distributed execution model. For example, an activity worker will somehow have to get custom tags from the workflow execution to fit your "regions" usecase (so, need to bring ContextPropagator here). We don't have plans to make it possible. Too large pain, customization, and risk of unsafety for a narrow use-case. Core worker metrics should be unified and simple. Giving each workflow execution an opportunity to configure a unique scope for each execution doesn't look like a good practice or a good idea. Hence, "won't fix". You can set your own scope in each workflow for your custom workflow metrics though.
What we currently have is we have a single WorkflowServiceStubs which we cannot use to set the region. We tried to create a WorkflowServiceStubs instance per region, but that starts to break as soon as we need to react to signals, we there loose the region and cannot use the proper one.
You can have separate task queues for each "region" and start separate workers for each of them. Each worker will be configured with its own task queue and its own MetricsScope. This way signals will be routed only to the worker attributed to the right task queue and region and will get correct tags on signals.
