opentelemetry-python icon indicating copy to clipboard operation
opentelemetry-python copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

Not able to auto-instrument Python application using gunicorn server

Open xwgao opened this issue 2 years ago • 10 comments

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

Environment:

  • OpenShift Container Platform (OCP) version: 4.12.22
  • Kubernetes version: v1.25.10+8c21020
  • Python3 version: Python 3.9.16

Steps to reproduce Describe exactly how to reproduce the error. Include a code sample if applicable.

  1. I installed Community OpenTelemetry Operator v0.89.0 in my OpenShift 4.12.22 cluster.
  2. I created an OpenTelemetry instrumentation as below.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  annotations:
    instrumentation.opentelemetry.io/default-auto-instrumentation-apache-httpd-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    instrumentation.opentelemetry.io/default-auto-instrumentation-dotnet-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-go-image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha
    instrumentation.opentelemetry.io/default-auto-instrumentation-java-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.31.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-nginx-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    instrumentation.opentelemetry.io/default-auto-instrumentation-nodejs-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.44.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-python-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.41b0
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: instrumentation
  namespace: my-namespace
spec:
  apacheHttpd:
    configPath: /usr/local/apache2/conf
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 1m
        memory: 128Mi
    version: "2.4"
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 128Mi
  exporter:
    endpoint: http://otel-collector-headless:4317
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 32Mi
      requests:
        cpu: 50m
        memory: 32Mi
  java:
    env:
    - name: OTEL_INSTRUMENTATION_LIBERTY_ENABLED
      value: "true"
    - name: OTEL_METRICS_EXPORTER
      value: none
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.31.0
    resources:
      limits:
        cpu: 500m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  nginx:
    configFile: /etc/nginx/nginx.conf
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 1m
        memory: 128Mi
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.44.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 128Mi
  propagators:
  - tracecontext
  - baggage
  - b3
  python:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector-headless:4318
    - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
      value: sqlite3
    - name: OTEL_PYTHON_LOG_CORRELATION
      value: "true"
    - name: OTEL_PYTHON_LOG_LEVEL
      value: debug
    - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
      value: "true"
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.41b0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 32Mi
      requests:
        cpu: 50m
        memory: 32Mi
  resource: {}
  sampler:
    argument: "1"
    type: parentbased_traceidratio
  1. I created an OpenTelemetry collector as below.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: otel
  namespace: my-namespace
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      metricstransform:
        transforms:
          - include: mas-core.duration
            match_type: regexp
            action: update
            operations:
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code

    exporters:
      logging:
        verbosity: detailed
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-core
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 4s, 6s, 8s, 10s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.url
          - name: http.route
          - name: http.host

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging]
        metrics:
          receivers: [spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging]
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.89.0
  ingress:
    route: {}
  managementState: managed
  mode: statefulset
  observability:
    metrics: {}
  podDisruptionBudget:
    maxUnavailable: 1
  replicas: 1
  resources: {}
  targetAllocator:
    prometheusCR:
      scrapeInterval: 30s
    resources: {}
  updateStrategy: {}
  upgradeStrategy: automatic
  1. I added below annotation into the deployment for my Python application which uses gunicorn server.
    instrumentation.opentelemetry.io/inject-python: "instrumentation
  1. As per https://github.com/open-telemetry/opentelemetry-python/issues/2038, I found that currently auto-instrumentation for Python app using gunicorn server is not supported yet. So I added the following code into gunicorn.conf.py of my Python application.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)
    service_name = os.getenv("OTEL_SERVICE_NAME")
    resource = Resource.create(attributes={
        "service.name": service_name
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint=otlp_endpoint)
    )
    trace.get_tracer_provider().add_span_processor(span_processor)
  1. Then after the pod container of my deployment started, I still can not get the traces exported to my otel collector.

What is the expected behavior? What did you expect to see?

The traces of my Python application can be exported to otel collector with auto-instrumentation.

What is the actual behavior? What did you see instead?

The traces of my Python application can not be exported to otel collector with auto-instrumentation.

Additional context Add any other context about the problem here.

xwgao avatar Dec 08 '23 09:12 xwgao

And I also found the following error messages from the pod log of my deployment.

[2023-12-08 06:58:08 +0000] [57] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/geventlet.py", line 143, in init_process
    super().init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/ibm/internalapi/internalapi.py", line 22, in <module>
    from api import usersAPI, workspacesAPI, applicationsAPI, metadataAPI, openidAPI, messageAPI, groupsAPI, datadictionaryAPI, manageworkspaceAPI, bindingsAPI, passwordPolicyAPI, idpsAPI
  File "/opt/ibm/internalapi/api/__init__.py", line 15, in <module>
    from api.metadata import metadataAPI
  File "/opt/ibm/internalapi/api/metadata.py", line 17, in <module>
    from managers import MetadataMgr
  File "/opt/ibm/internalapi/managers/__init__.py", line 15, in <module>
    from managers.datadictionary import DataDictionaryMgr
  File "/opt/ibm/internalapi/managers/datadictionary.py", line 41, in <module>
    k8sClient = k8sUtil.dynClient
  File "/usr/local/lib/python3.9/site-packages/mas/utils/k8s/k8sUtil.py", line 105, in dynClient
    self._dynClient = DynamicClient(k8s_client)
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/client.py", line 40, in __init__
    K8sDynamicClient.__init__(self, client, cache_file=cache_file, discoverer=discoverer)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
    self.__discoverer = discoverer(self, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 228, in __init__
    Discoverer.__init__(self, client, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 54, in __init__
    self.__init_cache()
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 70, in __init_cache
    self._load_server_info()
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/discovery.py", line 98, in _load_server_info
    'kubernetes': self.client.request('get', '/version', serializer=just_json)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    api_response = self.client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 110, in request
    return self.request_encode_url(
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 143, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/otel-auto-instrumentation-python/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/otel-auto-instrumentation-python/wrapt/wrappers.py", line 669, in __call__
    return self._self_wrapper(self.__wrapped__, self._self_instance,
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/urllib3/__init__.py", line 243, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 735, in _ssl_wrap_socket_and_match_hostname
    context = create_urllib3_context(
  File "/otel-auto-instrumentation-python/urllib3/util/ssl_.py", line 292, in create_urllib3_context
    context.minimum_version = TLSVersion.TLSv1_2
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  [Previous line repeated 457 more times]
RecursionError: maximum recursion depth exceeded
[2023-12-08 06:58:08 +0000] [57] [INFO] Worker exiting (pid: 57)
CPU_LIMITS for INTERNALAPI: 2
[2023-12-08 06:58:08 +0000] [62] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/geventlet.py", line 143, in init_process
    super().init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/ibm/internalapi/internalapi.py", line 22, in <module>
    from api import usersAPI, workspacesAPI, applicationsAPI, metadataAPI, openidAPI, messageAPI, groupsAPI, datadictionaryAPI, manageworkspaceAPI, bindingsAPI, passwordPolicyAPI, idpsAPI
  File "/opt/ibm/internalapi/api/__init__.py", line 15, in <module>
    from api.metadata import metadataAPI
  File "/opt/ibm/internalapi/api/metadata.py", line 17, in <module>
    from managers import MetadataMgr
  File "/opt/ibm/internalapi/managers/__init__.py", line 15, in <module>
    from managers.datadictionary import DataDictionaryMgr
  File "/opt/ibm/internalapi/managers/datadictionary.py", line 41, in <module>
    k8sClient = k8sUtil.dynClient
  File "/usr/local/lib/python3.9/site-packages/mas/utils/k8s/k8sUtil.py", line 105, in dynClient
    self._dynClient = DynamicClient(k8s_client)
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/client.py", line 40, in __init__
    K8sDynamicClient.__init__(self, client, cache_file=cache_file, discoverer=discoverer)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
    self.__discoverer = discoverer(self, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 228, in __init__
    Discoverer.__init__(self, client, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 54, in __init__
    self.__init_cache()
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 70, in __init_cache
    self._load_server_info()
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/discovery.py", line 98, in _load_server_info
    'kubernetes': self.client.request('get', '/version', serializer=just_json)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    api_response = self.client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 110, in request
    return self.request_encode_url(
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 143, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/otel-auto-instrumentation-python/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/otel-auto-instrumentation-python/wrapt/wrappers.py", line 669, in __call__
    return self._self_wrapper(self.__wrapped__, self._self_instance,
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/urllib3/__init__.py", line 243, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 735, in _ssl_wrap_socket_and_match_hostname
    context = create_urllib3_context(
  File "/otel-auto-instrumentation-python/urllib3/util/ssl_.py", line 292, in create_urllib3_context
    context.minimum_version = TLSVersion.TLSv1_2
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  [Previous line repeated 457 more times]
RecursionError: maximum recursion depth exceeded

xwgao avatar Dec 08 '23 09:12 xwgao

I am having similar issues with my setup. Instead of using the OTEL collector exporter I use the ConsoleSpanExporter and confirm there is no output in the console. Will update if I find a solution.

halkony avatar Feb 24 '24 18:02 halkony

I found a working solution, though it doesn't use the DjangoInstrumentor. OpenTelemetry doesn't play nice with ASGI, but there is an explicit ASGI middleware for open telemetry. I followed this article and got things working.

pip install opentelemetry-instrumentation-asgi

In gunicorn.conf.py

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)
    
    resource = Resource.create({"service.name": "django"})

    trace.set_tracer_provider(TracerProvider(resource=resource))
    trace.get_tracer_provider().add_span_processor(
        BatchSpanProcessor(ConsoleSpanExporter())
    )

In asgi.py (or whereever your asgi config is)

from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware
django_application = get_asgi_application()
django_application = OpenTelemetryMiddleware(django_application)

It's unclear whether these traces include the database transactions provided by DjangoInstrumentor, but I will look into it.

halkony avatar Feb 24 '24 18:02 halkony

I was able to get the telemetry working without the ASGI middleware package. You need to perform the instrumentation inside of the post fork handler per these docs.

import uptrace

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)
    
    uptrace.configure_opentelemetry(
        # dsn set as env var
        service_name="django",
        service_version="0.1.0",
    )
    DjangoInstrumentor().instrument()

You could replace the uptrace snippet in this code with the tracer snippets above for your specific backend.

Under the hood, DjangoInstrumentor().instrument() injects middleware in Django, and if done successfully you will see it in the middleware of the Django Debug Toolbar image

halkony avatar Feb 24 '24 21:02 halkony

Here is the implementation I ended up with:

 ## app/otel/sdk.py
 from os import environ, pathsep, getcwd
 from os.path import abspath, dirname
 import opentelemetry.instrumentation.auto_instrumentation
 
 # Optionally initialize OTel, by importing this file first
 if environ.get("OTEL_SDK_DISABLED") != "true":
     # Instrument packages through prefixed `PYTHONPATH` that includes instrumented packages first
     python_path = environ.get("PYTHONPATH")
 
     if not python_path:
         python_path = []
 
     else:
         python_path = python_path.split(pathsep)
 
     cwd_path = getcwd()
 
     # This is being added to support applications that are being run from their
     # own executable, like Django.
     # FIXME investigate if there is another way to achieve this
     if cwd_path not in python_path:
         python_path.insert(0, cwd_path)
 
     filedir_path = dirname(abspath(opentelemetry.instrumentation.auto_instrumentation.__file__))
 
     python_path = [path for path in python_path if path != filedir_path]
 
     python_path.insert(0, filedir_path)
 
     environ["PYTHONPATH"] = pathsep.join(python_path)
 
     # Initialize OTel components via ENV variables
     # (tracer provider, meter provider, logger provider, processors, exporters, etc.)
     from opentelemetry.instrumentation.auto_instrumentation import sitecustomize  # noqa: F401
 
 
 ## app/main.py
 import app.otel.sdk  # noqa: F401
 ...
 
 
 ## gunicorn.conf.py
 ...
 # NOTE: OTel does not work well with `gunicorn` preloading the app, if this is crucial,
 # consider using `preload_app = True` + [application factory](https://docs.gunicorn.org/en/stable/run.html)
 preload_app = False
 # removed the `post_fork` hook
 ...

Then running the application just like gunicorn app.main:app

In my case I am instrumenting within the code, not from the opentelemetry-operator and the code seems to works well. Yet I hope it will inspire you for some other approaches.

smoke avatar Feb 25 '24 08:02 smoke

@halkony I've changed my post fork handler as below, but still got the same error.

import uptrace
from opentelemetry.instrumentation.django import DjangoInstrumentor

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)

    uptrace.configure_opentelemetry(
        service_name=os.getenv("OTEL_SERVICE_NAME"),
        service_version="1.0.0",
    )
    DjangoInstrumentor().instrument()

Did you use Python auto-instrumentation with the OpenTelemetry operator? And how did you use the ConsoleSpanExporter instead of using the OTEL collector exporter?

Thanks a lot for your help.

xwgao avatar Feb 27 '24 15:02 xwgao

@smoke Many thanks to your help! I tried with your code, and I found the warning messages from my gunicorn server pod log as below. Is this working as expected? Can I ignore this warning? Thanks.

can't parse Uptrace DSN: either dsn option or UPTRACE_DSN is required (Uptrace is disabled)
can't parse Uptrace DSN: either dsn option or UPTRACE_DSN is required (Uptrace is disabled)
can't parse Uptrace DSN: either dsn option or UPTRACE_DSN is required (Uptrace is disabled)
can't parse Uptrace DSN: either dsn option or UPTRACE_DSN is required (Uptrace is disabled)
WARNING:uptrace.uptrace:can't parse Uptrace DSN: either dsn option or UPTRACE_DSN is required (Uptrace is disabled)

I still can not find traces exported to my OTEL collector. And I found that you set the following env vars to export traces & metrics to console . Where can you find the exported traces / metrics / logs? Can you find them from the pod log? Thanks a lot.

OTEL_TRACES_EXPORTER=console
OTEL_METRICS_EXPORTER=console
OTEL_LOGS_EXPORTER=console

xwgao avatar Feb 29 '24 10:02 xwgao

@xwgao nope, your code is not working right.

Using my code provides all that you need and you must not include and use the Uptrace package itself, this gives you the warnings you see and the Uptrace package does not respect / configure the relevant console exporters.

In order to use my code and send signals to console (pod / container logs) - you should use the relevant environment variables that you mentioned.

In order to send signals to Uptrace you need my code and change the environment variables accordingly https://uptrace.dev/get/opentelemetry-python.html#already-using-otlp-exporter

also change

OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

smoke avatar Feb 29 '24 10:02 smoke

@smoke I want to send traces / metrics to my OTEL collector (the endpoint is http://otel-collector-headless:4317), rather than console nor Uptrace. How can I set the env vars? Thanks a lot.

xwgao avatar Mar 01 '24 09:03 xwgao

@xwgao this is a list of environment variables, use those that are not commented out and exact values you should be using to get there

### OTel ###
# The default settings export telemetry (traces - supported, metrics - supported, logs - supported)
# from `api` -> `OTel Collector` 
# For general understanding @see https://opentelemetry.io/docs/languages/python/

# For environment variables @see https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

# Disable / Enable the OTel, enabled unless set to "true"
#OTEL_SDK_DISABLED=true

# OTel internal logging level, defaults to "info"
#OTEL_LOG_LEVEL=info
# on PROD it better be
#OTEL_LOG_LEVEL=warn

# Enable exponential histograms.
# NOTE: Not supported by python and triggers warning, but will be provided anyway for other language distributions
#OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=BASE2_EXPONENTIAL_BUCKET_HISTOGRAM

# Prefer delta temporality.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA

OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector-headless:4317"
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Enable gzip compression, this is enabled by default
#OTEL_EXPORTER_OTLP_COMPRESSION=gzip

# `OTEL_TRACES_EXPORTER` can be `console` | `otlp` or comma separated set of exporters to combine them e.g. `console, otlp`
OTEL_TRACES_EXPORTER=otlp

# `OTEL_METRICS_EXPORTER` can be `console` | `otlp` or comma separated set of exporters to combine them e.g. `console, otlp`
OTEL_METRICS_EXPORTER=otlp

# `OTEL_LOGS_EXPORTER` can be `console` | `otlp` or comma separated set of exporters to combine them e.g. `console, otlp`
OTEL_LOGS_EXPORTER=otlp

# Resource attributes - these define the thing producing the telemetry
# accepted values is comma separated list of Semantic Resource Attributes
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=local"

# Set the `service.name` resource attribute
OTEL_SERVICE_NAME="api"

# SemVer of this service, should be set by CI/CD
#OTEL_SERVICE_VERSION="3.2.1-rc.1"

### /OTel ###

smoke avatar Mar 01 '24 09:03 smoke