opentelemetry-operator icon indicating copy to clipboard operation
opentelemetry-operator copied to clipboard

auto-instrumentation: can't preserve ownership of initcontainer volume

Open leofvo opened this issue 1 year ago • 3 comments

Component(s)

instrumentation

What happened?

Description

I using python auto-instrumentation in a kubernetes cluster. I have installed the operator and the instrumentation CRD (all up to date). I have deployed about 6 services, all using fastapi. And one of my service isn't sending metrics and traces to my collector. The collector is the same, operator same, same namespace and the instrument CR too. image

Steps to Reproduce

Expected Result

Init container booting and finished without any issues..

Actual Result

When my pods is starting, the init container launch and this kind of error message appears: cp: can't preserve ownership of '/otel-auto-instrumentation-python/./zipp-3.17.0.dist-info/top_level.txt': Operation not permitted

I can't get traces or metrics exported to my collector...

Here is a screenshot of the initcontainer logs:

image

Kubernetes Version

v1.28.5-eks-5e0fdde

Operator version

v0.93.0

Collector version

v0.90.1

Environment information

││     app.kubernetes.io/component: controller-manager
││     app.kubernetes.io/instance: opentelemetry-operator
││     app.kubernetes.io/managed-by: Helm
││     app.kubernetes.io/name: opentelemetry-operator
││     app.kubernetes.io/version: 0.93.0
││     argocd.argoproj.io/instance: observability
││     helm.sh/chart: opentelemetry-operator-0.47.0

Log output

No response

Additional context

No response

leofvo avatar Feb 12 '24 16:02 leofvo

Update: I tried to upgrade the volume size limit, still having the issue.

It doesn't seem to be related to the available storage limit.

leofvo avatar Feb 13 '24 09:02 leofvo

Can you post the application's Deployment manifest?

I've faced the same issue with the Node.js instrumentation, which I've been able to pinpoint to container settings runAsNonRoot: true (this gets preserve ownership errors) vs runAsNonRoot: false (everything works).

illrill avatar Feb 15 '24 10:02 illrill

I had runAsNonRoot: true, I removed it and now my traces are collected, but still having the error message in my initContainer logs. Thanks for the help !

leofvo avatar Feb 19 '24 11:02 leofvo

this issue is also related my one #2726 . it looks like it doesn't affect the collecting trace with these message, see #1013 , but still need a proper way to avoid.

smallc2009 avatar Mar 07 '24 03:03 smallc2009

I would say #2695 should fix this issue

iblancasa avatar Mar 07 '24 14:03 iblancasa