opentelemetry-python-contrib
opentelemetry-python-contrib copied to clipboard
Django app on K8s crashes when auto-instrumented via the opentelemetry-operator without explicitly setting the PYTHONPATH and DJANGO_SETTINGS_MODULE
Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.
- Python version: 3.11
- Platform: Kubernetes
- Installed dependencies:
- Django~=5.0.4
- requests==2.31.0
- Hosting: hosting locally on minikube, but same issue when hosted on AWS EKS
Steps to reproduce Sample app code is here: https://github.com/srprash/otel-python-k8s-samples/tree/main/django
- Install Docker and Minikube, and run both.
- Run
eval $(minikube docker-env)
so that minikube can use the docker images from local repository. - Build the application docker image using
docker build -t django-app .
- Deploy the application to minikube using
kubectl apply -f k8s/deployment.yaml
andkubectl apply -f k8s/service.yaml
- Enable port-forwarding
kubectl port-forward svc/django-service 8000
- Make a request to the application on
http://127.0.0.1:8000/outgoing-http-call
endpoint
What is the expected behavior? What did you expect to see?
- The application should run without an issues and produce a server span for the incoming request and a client span for the outgoing http request
What is the actual behavior? What did you see instead?
- The application fails to start with the error:
Defaulted container "django-app" out of: django-app, opentelemetry-auto-instrumentation-python (init) CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.
- When I uncomment these lines (that is, explicitly set the
PYTHONPATH
andDJANGO_SETTINGS_MODULE
), only then the application works correctly and is instrumented.
Additional context
- Another related issue: https://github.com/open-telemetry/opentelemetry-operator/issues/2302
- Note that I have another application using Flask and this issue is not seen there.
- The issue with Django is specifically when using the opentelemetry-operator to do the auto-instrumentation by adding the
instrumentation.opentelemetry.io/inject-python: "true"
annotation to the application deployment manifest. Auto-instrumentation via theopentelemetry-instrument
command works fine (likely due to this code that handles the current working directory).