@Timed annotation with dynamic tags derived from input arguments and/or output

[kosty]

Use case

While @Timed annotation provides good starting point for annotating particular methods in a single service. In case of a platform single API can get used by completely different clients with somewhat different access patterns. Implies different notification thresholds or alert receivers. Consider a sample api

    @Timed
    public Response process(BucketId bucketId, ProcessingParams params) { ... }

If all incoming requests are timed in a same manner a metric visualization could take the form of

However if we were to take a look at it with a bit more precision, say plot requests to different bucket ids separately one could arrive at

Going further, excluding outliers paints a completely different picture

As it usually happens, some like 20% of requests of specific form could incur up to 80% of support burden and identifying those would make the whole monitoring exercise much more useful.

Proposal

Introduce a new (or extend existing?) @Timed annotation which could take optional list of tag names as well as custom expressions similar to SpEL

So that annotations could take form of

    @Timed(
       inputTags = {"bucket_id", "#{[0]}"},
       outputTags = {"status": "#{[0]?.status}"}
    )
    public Response process(BucketId bucketId, ProcessingParams params) { ... }

Cardinality

This, of course, is open to some abuse in cases where cardinality of input/output values becomes high. Consider what would happen in example above if bucket_id turns out to be a randomly generated UUIDv4. In such a case, making sure the right value/expression is provided is ultimately part of developer's responsibility.

It is worth noting at this point the current alternative is to provide a dedicated endpoint to every specific use-case which does not seem scalable.

References

This issue has use case similar to #1586 yet a different approach to address it. This issue is different from extraTags in a sense that extraTags are static and current issue proposes dynamically evaluated values. This issue is different from #1265 in a sense that it is protocol agnostic. It is also different from #535 in a sense that tag value is provided "declaratively"

[kosty]

Another approach could be configuration of a @Timed annotation, so that extraTags were dynamically evaluated. Similar to

    @Timed(
       extraTags = {"bucket_id", "#{bucketId}", "status": "#{result?.status}"}
    )
    public Response process(BucketId bucketId, ProcessingParams params) { ... }

This however implies there should be a way to dynamically associate parameters with their names

[michaelmcfadyen]

Micrometer is decoupled from spring so I'm not sure that using SpEL is a suitable solution.

[codefromthecrypt]

this concern came up in a library I worked on in the past, Feign. It is common to any annotation based modeling library I think. What we used instead of adding things to the annotation itself, is a helper. This allows you to do whatever you want, including a spring specific implementation if available. For example a null "expander" or "extractor" could defer to a default or a platform based lookup

https://github.com/OpenFeign/feign/blob/master/core/src/main/java/feign/Param.java#L36

https://github.com/OpenFeign/feign/blob/d3f7d76f57edc07cc5008cd01d0477effe90b0c3/README.md#custom-expansion

maybe this is useful?

[ursvasan]

@kosty I have a similar requirement and wondering what solution you went with. I am working with a multi tenant system where i want to add common tags (that i know at deployment time - stack, region, etc) and some dynamic tags (that i know only during request time - including tenant_id, etc). It need not be an annotation based solution. I am currently thinking of a Threadlocal based solution. But curious to know what is the recommended approach.

[misrasuraj]

I have also same requirement. We have multi-tenant application and we need to add the tenant information as tag while sending the histogram information through micrometer. Please let me know if there is any recommended approach for that.

[ursvasan]

@misrasuraj I just ended up using the helper methods instead of annotation approach. You can find the official documentation here.

[kevinkraus]

This feature would be very useful for me, please let me know what I can do to help.

[jonatan-ivanov]

Sorry for not getting back to this earlier. As it was called out in https://github.com/micrometer-metrics/micrometer/issues/1732#issuecomment-573325853, since Micrometer does not depend on Spring and since it is used in other frameworks too, using SpEL is not feasible. Using something like JXPath I think is possible but I also think that this solution is quite brittle since you lose the support of the compiler.

What do you think about the solution that Adrian proposed in https://github.com/micrometer-metrics/micrometer/issues/1732#issuecomment-573367184? I created a PoC for that in https://github.com/micrometer-metrics/micrometer/pull/2121/files. I think it should give you the similar experience but in a type-checked way. It can be used to add static tags:

@Timed
void testMethod(@MetricTag("testTag") String param) {...}

or dynamic ones with a value mapper:

@Timed
void testMethod(@MetricTag(key="testTag", valueMapper=YourValueMapper.class) String param) {...}

[mad0gre]

Good to know this is being worked on. The @MetricTag approach would certainly cover this use case.

[marcingrzejszczak]

I've created a PR to solve this issue https://github.com/micrometer-metrics/micrometer/pull/3727 . Feedback is more than welcome

[maciejwalkowiak]

While @MeterTag solves the part of passing tags from input arguments, as far as I can see there is no out-of-the-box way to pass tags coming from the method return value. Am I missing something or perhaps there is a plan to add it?

[jonatan-ivanov]

We don't have such a feature but it might worth an issue. I would be also curious what would that look like (e.g.: @MeterTag on the method so that it captures the result).

I also feel it is much more of a slippery slope from the point of view of high cardinality issues than @MeterTag on params. I.e.: you control the parameters you pass to the method but you might not control the result so you either need to wrap the original method and normalize the result there or you would do it in the ValueResolver (could be true for params as well though). Or, just use a Timer without an annotation. Don't get me wrong, I'm not against it, I did this in the past (not with Micrometer).