opentelemetry-specification icon indicating copy to clipboard operation
opentelemetry-specification copied to clipboard

Q: Way to ignore healthcheck traces when using automatic tracer across all languages?

Open naseemkullah opened this issue 5 years ago • 12 comments

Hi,

Will there be a way to ignore specific endpoints such as health checks when using the automatic tracer?

naseemkullah avatar Jul 04 '19 22:07 naseemkullah

Hi, not sure I understand the question correctly. Do you think of a way to exclude e.g. specific URLs? Why only for automatic tracer?

z1c0 avatar Jul 09 '19 14:07 z1c0

Yes it would be nice to exclude specific URLs in the automatic tracer, and also not in the automatic tracer, but just want to make sure this exclusion feature is configurable in the automatic tracer.

naseemkullah avatar Jul 09 '19 14:07 naseemkullah

I agree that URL exclusions would make sense. We currently have that feature in Dynatrace for exactly that purpose.

In OpenTelementry, the broader question is:

At what level does such configuration need to be applied?

  • API (config is exposed via API level)
  • Tracer-impl (every tracer-implementation can have it's own config system)
    • Should the OTel SDK have such a config system?
  • Exporter (it's up to the exporter to have a config)

I think config should not be exposed via API. Would add much more complexity. With healthchecks specifically however, it could make sense that a user gives a "hint" about "unimportant" requests (such as healthchecks), so that the Tracer-impl can decide on it's own to exclude them or not.

discostu105 avatar Jul 22 '19 07:07 discostu105

+1 to this issue. Some OpenCensus implementations had a list of paths that wouldn't get sampled, which were often used to ensure that health checks, exporters, and other telemetry providers weren't generating traces. This was incredibly useful.

mtwo avatar Jul 22 '19 15:07 mtwo

This feels like an SDK detail, but it would be great if there was a way to do this externally: a config file for example.

tedsuo avatar Dec 04 '19 17:12 tedsuo

https://github.com/open-telemetry/opentelemetry-js/issues/585 might be of interest

naseemkullah avatar Dec 04 '19 17:12 naseemkullah

Has been any decision made on this? I have started looking into this in the context of https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060

From the thread I cannot figure out how the skip URL pattern should work. There are two approaches

  1. exclude a span from being created
  2. create non-recording (not sampled) span, but respect upstream sampling decision

The first approach might result in broken traces (e.g. a health check might have an upstream parent or it calls any downstream service/instrumentation). The second approach is more consistent.

Is there a way to instruct the tracer to create non-recording span?

pavolloffay avatar Sep 24 '20 08:09 pavolloffay

from @iNikem

I think the right way is to use one of the factory methods on io.opentelemetry.trace.DefaultSpan.

So there is a way how to create non-recording span. The DefaultSpan has invalid context with zero span ID. I don't like two things about this:

  • two code paths for creating a span
  • invalid context with zero IDs. I would prefer a valid non-sampled context.

In OpenTracing there was sampling.priority=integer tag that span builder could use for sampling decision. This way multiple code paths were avoided.

pavolloffay avatar Sep 24 '20 08:09 pavolloffay

I guess we could implement a custom Sampler? It could do ParentOrElse for non-matched paths and ParentOnly for marched paths I think.

DefaultSpan is going to be renamed to something like PropagatorOnlySpan so it'll become less appropriate, or we need to otherwise chime in on the spec issue that we have a valid use case here.

anuraaga avatar Sep 24 '20 10:09 anuraaga

Forgive me if this is not in the right place, but I think I have a related usecase... I'm starting to use opentelemetry, and I've noticed that some of my libraries (go-redis) create traces. I can see how this is powerful, but right now, they're just clutter. So I'd love a reasonably low-level way to exclude them. I imagine by name, but I'm a bit fuzzy on all the nomenclature.

I think the best I can do right now, is configs in my exporter, but that feels very cumbersome.

directionless avatar Oct 12 '20 04:10 directionless

@carlosalberto I think this is a duplicate of #1597 and should be closed.

iNikem avatar Aug 03 '21 15:08 iNikem

@iNikem I want to add that #1597 is comparable, but in this case quite different. They are the other way around for each other and both valid use cases.

Anyway, do we already have a solution to this issue?

xvilo avatar Dec 14 '23 10:12 xvilo