opentelemetry-python-contrib icon indicating copy to clipboard operation
opentelemetry-python-contrib copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

Not able to auto-instrument Python application using gunicorn server

Open xwgao opened this issue 1 year ago • 6 comments
trafficstars

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

Environment:

  • OpenShift Container Platform (OCP) version: 4.12.22
  • Kubernetes version: v1.25.10+8c21020
  • Python3 version: Python 3.9.16

Steps to reproduce Describe exactly how to reproduce the error. Include a code sample if applicable.

  1. I installed Community OpenTelemetry Operator v0.89.0 in my OpenShift 4.12.22 cluster.
  2. I created an OpenTelemetry instrumentation as below.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  annotations:
    instrumentation.opentelemetry.io/default-auto-instrumentation-apache-httpd-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    instrumentation.opentelemetry.io/default-auto-instrumentation-dotnet-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-go-image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.8.0-alpha
    instrumentation.opentelemetry.io/default-auto-instrumentation-java-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.31.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-nginx-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    instrumentation.opentelemetry.io/default-auto-instrumentation-nodejs-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.44.0
    instrumentation.opentelemetry.io/default-auto-instrumentation-python-image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.41b0
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: instrumentation
  namespace: my-namespace
spec:
  apacheHttpd:
    configPath: /usr/local/apache2/conf
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 1m
        memory: 128Mi
    version: "2.4"
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 128Mi
  exporter:
    endpoint: http://otel-collector-headless:4317
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.1.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 32Mi
      requests:
        cpu: 50m
        memory: 32Mi
  java:
    env:
    - name: OTEL_INSTRUMENTATION_LIBERTY_ENABLED
      value: "true"
    - name: OTEL_METRICS_EXPORTER
      value: none
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.31.0
    resources:
      limits:
        cpu: 500m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  nginx:
    configFile: /etc/nginx/nginx.conf
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 1m
        memory: 128Mi
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.44.0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 128Mi
  propagators:
  - tracecontext
  - baggage
  - b3
  python:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector-headless:4318
    - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
      value: sqlite3
    - name: OTEL_PYTHON_LOG_CORRELATION
      value: "true"
    - name: OTEL_PYTHON_LOG_LEVEL
      value: debug
    - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
      value: "true"
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.41b0
    resourceRequirements:
      limits:
        cpu: 500m
        memory: 32Mi
      requests:
        cpu: 50m
        memory: 32Mi
  resource: {}
  sampler:
    argument: "1"
    type: parentbased_traceidratio
  1. I created an OpenTelemetry collector as below.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: otel
  namespace: my-namespace
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      metricstransform:
        transforms:
          - include: mas-core.duration
            match_type: regexp
            action: update
            operations:
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code

    exporters:
      logging:
        verbosity: detailed
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-core
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 4s, 6s, 8s, 10s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.url
          - name: http.route
          - name: http.host

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging]
        metrics:
          receivers: [spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging]
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.89.0
  ingress:
    route: {}
  managementState: managed
  mode: statefulset
  observability:
    metrics: {}
  podDisruptionBudget:
    maxUnavailable: 1
  replicas: 1
  resources: {}
  targetAllocator:
    prometheusCR:
      scrapeInterval: 30s
    resources: {}
  updateStrategy: {}
  upgradeStrategy: automatic
  1. I added below annotation into the deployment for my Python application which uses gunicorn server.
    instrumentation.opentelemetry.io/inject-python: "instrumentation
  1. As per https://github.com/open-telemetry/opentelemetry-python/issues/2038, I found that currently auto-instrumentation for Python app using gunicorn server is not supported yet. So I added the following code into gunicorn.conf.py of my Python application.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)
    service_name = os.getenv("OTEL_SERVICE_NAME")
    resource = Resource.create(attributes={
        "service.name": service_name
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint=otlp_endpoint)
    )
    trace.get_tracer_provider().add_span_processor(span_processor)
  1. Then after the pod container of my deployment started, I still can not get the traces exported to my otel collector.

What is the expected behavior? What did you expect to see?

The traces of my Python application can be exported to otel collector with auto-instrumentation.

What is the actual behavior? What did you see instead?

The traces of my Python application can not be exported to otel collector with auto-instrumentation.

Additional context And I also found the following error messages from the pod log of my deployment.

[2023-12-08 06:58:08 +0000] [57] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/geventlet.py", line 143, in init_process
    super().init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/ibm/internalapi/internalapi.py", line 22, in <module>
    from api import usersAPI, workspacesAPI, applicationsAPI, metadataAPI, openidAPI, messageAPI, groupsAPI, datadictionaryAPI, manageworkspaceAPI, bindingsAPI, passwordPolicyAPI, idpsAPI
  File "/opt/ibm/internalapi/api/__init__.py", line 15, in <module>
    from api.metadata import metadataAPI
  File "/opt/ibm/internalapi/api/metadata.py", line 17, in <module>
    from managers import MetadataMgr
  File "/opt/ibm/internalapi/managers/__init__.py", line 15, in <module>
    from managers.datadictionary import DataDictionaryMgr
  File "/opt/ibm/internalapi/managers/datadictionary.py", line 41, in <module>
    k8sClient = k8sUtil.dynClient
  File "/usr/local/lib/python3.9/site-packages/mas/utils/k8s/k8sUtil.py", line 105, in dynClient
    self._dynClient = DynamicClient(k8s_client)
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/client.py", line 40, in __init__
    K8sDynamicClient.__init__(self, client, cache_file=cache_file, discoverer=discoverer)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
    self.__discoverer = discoverer(self, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 228, in __init__
    Discoverer.__init__(self, client, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 54, in __init__
    self.__init_cache()
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 70, in __init_cache
    self._load_server_info()
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/discovery.py", line 98, in _load_server_info
    'kubernetes': self.client.request('get', '/version', serializer=just_json)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    api_response = self.client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 110, in request
    return self.request_encode_url(
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 143, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/otel-auto-instrumentation-python/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/otel-auto-instrumentation-python/wrapt/wrappers.py", line 669, in __call__
    return self._self_wrapper(self.__wrapped__, self._self_instance,
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/urllib3/__init__.py", line 243, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 735, in _ssl_wrap_socket_and_match_hostname
    context = create_urllib3_context(
  File "/otel-auto-instrumentation-python/urllib3/util/ssl_.py", line 292, in create_urllib3_context
    context.minimum_version = TLSVersion.TLSv1_2
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  [Previous line repeated 457 more times]
RecursionError: maximum recursion depth exceeded
[2023-12-08 06:58:08 +0000] [57] [INFO] Worker exiting (pid: 57)
CPU_LIMITS for INTERNALAPI: 2
[2023-12-08 06:58:08 +0000] [62] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/geventlet.py", line 143, in init_process
    super().init_process()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/ibm/internalapi/internalapi.py", line 22, in <module>
    from api import usersAPI, workspacesAPI, applicationsAPI, metadataAPI, openidAPI, messageAPI, groupsAPI, datadictionaryAPI, manageworkspaceAPI, bindingsAPI, passwordPolicyAPI, idpsAPI
  File "/opt/ibm/internalapi/api/__init__.py", line 15, in <module>
    from api.metadata import metadataAPI
  File "/opt/ibm/internalapi/api/metadata.py", line 17, in <module>
    from managers import MetadataMgr
  File "/opt/ibm/internalapi/managers/__init__.py", line 15, in <module>
    from managers.datadictionary import DataDictionaryMgr
  File "/opt/ibm/internalapi/managers/datadictionary.py", line 41, in <module>
    k8sClient = k8sUtil.dynClient
  File "/usr/local/lib/python3.9/site-packages/mas/utils/k8s/k8sUtil.py", line 105, in dynClient
    self._dynClient = DynamicClient(k8s_client)
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/client.py", line 40, in __init__
    K8sDynamicClient.__init__(self, client, cache_file=cache_file, discoverer=discoverer)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
    self.__discoverer = discoverer(self, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 228, in __init__
    Discoverer.__init__(self, client, cache_file)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 54, in __init__
    self.__init_cache()
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py", line 70, in __init_cache
    self._load_server_info()
  File "/usr/local/lib/python3.9/site-packages/openshift/dynamic/discovery.py", line 98, in _load_server_info
    'kubernetes': self.client.request('get', '/version', serializer=just_json)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 55, in inner
    resp = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/client.py", line 270, in request
    api_response = self.client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 110, in request
    return self.request_encode_url(
  File "/otel-auto-instrumentation-python/urllib3/_request_methods.py", line 143, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/otel-auto-instrumentation-python/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/otel-auto-instrumentation-python/wrapt/wrappers.py", line 669, in __call__
    return self._self_wrapper(self.__wrapped__, self._self_instance,
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/urllib3/__init__.py", line 243, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/otel-auto-instrumentation-python/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/otel-auto-instrumentation-python/urllib3/connection.py", line 735, in _ssl_wrap_socket_and_match_hostname
    context = create_urllib3_context(
  File "/otel-auto-instrumentation-python/urllib3/util/ssl_.py", line 292, in create_urllib3_context
    context.minimum_version = TLSVersion.TLSv1_2
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.9/ssl.py", line 587, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  [Previous line repeated 457 more times]
RecursionError: maximum recursion depth exceeded

xwgao avatar Dec 12 '23 08:12 xwgao

see https://github.com/open-telemetry/opentelemetry-python-contrib/issues/385#issuecomment-808792269

Badrmoh avatar Dec 16 '23 23:12 Badrmoh

@Badrmoh I've already imported the sitecustomize, my code is as below. But still got the same result. I also tried removing the other lines from post_fork (only reserve the first line which imports sitecustomize), the result is the same.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def post_fork(server, worker):
    from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
    server.log.info("Worker spawned (pid: %s)", worker.pid)
    service_name = os.getenv("OTEL_SERVICE_NAME")
    resource = Resource.create(attributes={
        "service.name": service_name
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint=otlp_endpoint)
    )
    trace.get_tracer_provider().add_span_processor(span_processor)

xwgao avatar Dec 18 '23 02:12 xwgao

@Badrmoh Is there any update? Thanks a lot.

xwgao avatar Dec 19 '23 12:12 xwgao

Sorry, I am not a maintainer. Hope you solve it.

Badrmoh avatar Dec 19 '23 13:12 Badrmoh

Find some related configuration instructions https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/instrument/python/ https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/instrument/python/gunicorn/ py code in above doc

import logging
from uuid import uuid4

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
    OTLPLogExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
# support for logs is currently experimental
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.resources import SERVICE_INSTANCE_ID
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# your gunicorn config here
# bind = "127.0.0.1:8000"

collector_endpoint = "http://localhost:4317"


def post_fork(server, worker):
    server.log.info("Worker spawned (pid: %s)", worker.pid)

    resource = Resource.create(
        attributes={
            # each worker needs a unique service.instance.id to distinguish the created metrics in prometheus
            SERVICE_INSTANCE_ID: str(uuid4()),
            "worker": worker.pid,
        }
    )

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=collector_endpoint)))
    trace.set_tracer_provider(tracer_provider)

    metrics.set_meter_provider(
        MeterProvider(
            resource=resource,
            metric_readers=[(PeriodicExportingMetricReader(
                OTLPMetricExporter(endpoint=collector_endpoint)
            ))],
        )
    )

    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=collector_endpoint)))
    logging.getLogger().addHandler(LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider))

lzpfmh avatar Jul 23 '24 12:07 lzpfmh

@lzpfmh Thanks a lot for your comments. I tried with your code, the only difference is that I set the value collector_endpoint to be http://otel-collector-headless:4317 which is the address of my OTEL collector service. But there is no traces exported to OTEL collector. Thanks.

xwgao avatar Jul 24 '24 04:07 xwgao