opentelemetry-python
opentelemetry-python copied to clipboard
Auto-instrumentation honouring the `OTEL_PYTHON_X_PROVIDER` env-vars
Is your feature request related to a problem?
I am auto-instrumenting a process in a container in kubernetes, and have this in my Dockerfile:
ENTRYPOINT [ "poetry", "run", "opentelemetry-instrument", "uvicorn", "platform_service.app:create_app", "--factory" ]
Sometimes I want to disable the instrumentation, and rather than override the ENTRYPOINT
or CMD
of the container from the Deployment (which requires copying the other parameters out into the kubernetes resource without opentelemetry-instrument
) or adding additional complexity into the Dockerfile, I'd like an env-var to disable the auto-instrumentation.
Describe the solution you'd like
As it happens, there are appropriate env-vars: OTEL_PYTHON_TRACER_PROVIDER
, OTEL_PYTHON_METER_PROVIDER
, and OTEL_PYTHON_LOG_EMITTER_PROVIDER
, which I would happily set to default
, to use the NoOpXProvider
entry-points from opentelemetry-api, rather than the default behaviour (equivalent to sdk
from the opentelemetry-sdk entrypoints.
However, there's two issues there:
- There's no
NoOpLogEmitterProvider
or even an ABC forLogEmitterProvider
. This would need to be added to the opentelemetry-sdk package, since it's not part of the OpenTelemetry API. -
_OTelSDKConfigurator._configure
calls_initialize_components
, and the functions the latter calls do not honour the*_PROVIDER
env-vars as they callset_X_provider
directly. The env-vars are processed byget_X_provider
ifset_X_provider
has not already been called.
The latter point is interesting as, because of the way opentelemetry-instrument
works, it offers those env-vars as command-line options. (Well, not the logger, right now, as the variable starts with _
, as did the metrics env-var before 1.12).
Describe alternatives you've considered
- Copying the command-line to run without OpenTelemetry into the kubernetes Deployment.
- Generating a separate container image with stripped command-line, so that only the default
ProxyXProviders
are loaded. (I can then use the env-vars to specifydefault
as described above. - Adding my own
BaseDistro
implementation that implements support forOTEL_PYTHON_X_PROVIDER
, and using that instead.- Technically, what I would be adding is an
_OTelSDKConfigurator
subclass, as that's the thing that triggers_initialize_components
. There's no actual tie between a distro entry point and a configurator entry point, soDefaultDistro
would be fine as-is. - If I was doing that, I'd probably ignore
OTEL_PYTHON_X_PROVIDER
and just provide a big "Disable everything" switch instead which skips all setup.
- Technically, what I would be adding is an
- Implementing the documented
none
values for theOTEL_X_EXPORTER
env-var and using those to disable export instead.- Edit: This is implemented, but not as entry-points. It's handled directly in the exporter auto-setup code that processes those env-vars. This is working for me locally with v1.11.1.
- Adding a command-line and env-var to
opentelemetry-instrument
which disables auto-instrumentation completely, somewhat along the lines ofOTEL_PYTHON_DISABLED_INSTRUMENTATIONS
but earlying-out ofinitialize
insitecustomize.py
.- For logging, I see that
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
has just been added by #2728 (and which defaults to false, and isn't visible inopentelemetry-instrumentation
help), which has a similar effect, but inside_initialize_components
. So maybeOTEL_PYTHON_AUTO_INSTRUMENTATION_ENABLED
is the generalised version of that.
- For logging, I see that
-
OpenTelemetryDistro
should not setOTEL_TRACES_EXPORTER
if it is not already set, but leave that to the user, as it does for logs and metrics, although with stabilisation of metrics in v1.12, I guess that'll change to enableOTEL_METRICS_EXPORTER
instead.- I ended up doing this, but in-passing for creating a project-local OTel Distribution library as a temporary fix for other things.
Additional context
I also noticed an odd issue that the get_X_provider
code is set up to use default_X
if the environment variable is not set, but the callers all ensure the environment variable is set before calling it. Perhaps the default_X
entrypoints should point at the ProxyXProvider
(except for logging, where that doesn't make sense, and since the code lives in the SDK, it can declare its own provider as default
), and a noop_X
entry-point could be used for the no-op providers explicitly, then we could always call opentelemetry.util._providers._load_provider
and let its env-var handling do the heavy lifting.
It also might make sense to make _load_provider
take a default provider to use if the env-var is not set, rather than falling back to default_X
if the env-var is not set. Or just remove the "env-var not set" support from _load_provider
.
~~So I'm not sure why, but it looks like if I don't provide the OTEL_X_EXPORTER
env-vars to the project, I don't get any errors, so I assume it's used the (in-effect) none
option, even those grpc
option is documented as the default.~~
~~It's doubly-weird because I have opentelemetry-distro
installed, and its OpenTelemetryDistro
should be being chosen and setting OTEL_TRACES_EXPORTER
if not already set.~~
~~It's possible that the exporters are just silent if there's nothing listening for them, I guess. I'm not sure if that's a problem or not, but it means this issue isn't a problem for me, and we can consider that one of the alternatives.~~
Edit: Gah. Turns out that #2594 was hiding the errors from opentelemetry.exporter.otlp.proto.grpc.exporter
in this case. So this isn't a solution.
I did try removing opentelemetry-distro
since my understanding of the auto-instrumentation code was that if I did that, it'd fall back to DefaultDistro
and hence honour the OTEL_X_EXPORTER
env-vars I am passing to the project, but it didn't work: without opentelemetry-distro
's opentelemetry_configurator
entry point, nothing causes _OTelSDKConfigurator
to be instantiated, hence _initialize_components
is never called.
Looking at the code while I was implementing a custom distro package, I've just noticed that "none"
option for the OTEL_X_EXPORTER
env-vars is supported, it's just handled directly in the auto-instrumentation code, rather than being an entrypoint. I'd only looked in setup.cfg for it, which is an oversight on my part.