OTEL tracing auth configuration problems

[wrighting]

If METAFLOW_OTEL_ENDPOINT is set and neither METAFLOW_SERVICE_AUTH_KEY or METAFLOW_SERVICE_HEADERS are set, then you get a warning message WARNING: no auth settings for Opentelemetry followed by a load of errors as it continues

self.span_exporter.shutdown()
AttributeError: 'NoneType' object has no attribute 'shutdown'

We use METAFLOW_SERVICE_AUTH_KEY for connecting to the metaflow service so it's set, when using the service, but doesn't apply to the OTEL endpoint (at the moment we don't have auth on OTEL, but if we do then it will be a different key)

[chriselion]

Yeah, I think the configuration would benefit from a separate header variable (equivalent to OTEL_EXPORTER_OTLP_HEADERS).

I think we've settled on monkey-patching metaflow.tracing.tracing_modules.init_tracing() so that it bypasses the TracerProvider setup and just uses our own shared library code to configure that.

[chriselion]

Full trace on the error spam:

Exception while exporting Span batch.
Traceback (most recent call last):
  File "/app/lib/python3.10/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 367, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
AttributeError: 'NoneType' object has no attribute 'export'
Exception ignored in atexit callback: <bound method TracerProvider.shutdown of <opentelemetry.sdk.trace.TracerProvider object at 0x7f37866ba5f0>>
Traceback (most recent call last):
  File "/app/lib/python3.10/site-packages/opentelemetry/sdk/trace/__init__.py", line 1248, in shutdown
    self._active_span_processor.shutdown()
  File "/app/lib/python3.10/site-packages/opentelemetry/sdk/trace/__init__.py", line 173, in shutdown
    sp.shutdown()
  File "/app/lib/python3.10/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 412, in shutdown
    self.span_exporter.shutdown()
AttributeError: 'NoneType' object has no attribute 'shutdown'

I think the problem is that set_otel_exporter() returns None here https://github.com/Netflix/metaflow/blob/9989bc6a31fe57524c2de19bddc458bba431a5b9/metaflow/tracing/span_exporter.py#L43-L45 which leads to None being passed here https://github.com/Netflix/metaflow/blob/9989bc6a31fe57524c2de19bddc458bba431a5b9/metaflow/tracing/tracing_modules.py#L42-L43