opentelemetry-operator
opentelemetry-operator copied to clipboard
open-telemetry
Not getting trace for all java and node js microservices
Describe the bug Not getting trace for all java and node js microservices
Steps to reproduce
i am patching the java services as below:
kubectl patch deploy/role1 -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'
kubectl patch deploy/role1 -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "true"}}}}}'
Same way I have patched another java service. for one service i am getting complete traces as well as I can see that service in AWS Opensearch > [Observability]>[Trace analytics]>[Services]. But for another service neither the service name or trace is populated in opensearch dashboard.
What did you expect to see? I want to see all the services which has been patched like above mentioned steps.
What did you see instead? another service neither the service name or trace is populated in opensearch dashboard.
What version did you use? image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.90.0 - manager image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 - rbac
What config did you use? apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: auto-instrumentation spec: exporter: endpoint: "http://observability-collector.aws-otel-eks.svc.cluster.local:4317/" propagators:
- tracecontext
- baggage
- b3 sampler: type: parentbased_traceidratio argument: "0.25" java: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest nodejs: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest python: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
Environment AWS EKS 1.27
Additional context NodeJS: kubectl patch deploy/management -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-nodejs": "true"}}}}}' kubectl patch deploy/management -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "true"}}}}}'
@shishir84 are the app being annotated and the Instrumentation CR in the same namespace? If not your annotation will not work. You'll see a log in the Operator logs saying something like "No Instrumentation resource found"
Yes the microservice and the auto-instrumentation is in same namespace.
i have 20+ java based microservices. all are implemented in java 11. when i have annotated all 20+ services, i am only getting traces from 6 microservices. Not sure why is it happening. I tried this in multiple environment like dev, stage and prod. everywhere same issue.
On the pods that are not working do you see the initContainer the operator injects for auto-instrumentation? You can see an example by looking at the pods that are working.
i have 20+ java based microservices. all are implemented in java 11. when i have annotated all 20+ services, i am only getting traces from 6 microservices. Not sure why is it happening. I tried this in multiple environment like dev, stage and prod. everywhere same issue.
Can you explain a little bit more about "everywhere same issue"? I mean: are there always exactly 6 applications sending traces? Are the same 6 applications?
Do you see any errors in the operator logs?
