aws-otel-collector
aws-otel-collector copied to clipboard
aws-observability
Support 128bit traceId on awsxray exporter
Is your feature request related to a problem? Please describe.
Trying to store traces into awsxray that use 128BIT traceId triggers an error:
│ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:54 TracesExporter {"kind": "exporter", "name": "awsxray", "type": "awsxray", "name": "awsxray", "#spans": 6} ││ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 0000000000000000b487e073065f48bf"} │
│ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 0000000000000000b487e073065f48bf"} ││ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 000000000000000078181036d7d2e51b"} │
│ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 000000000000000078181036d7d2e51b"} ││ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 00000000000000008c21d3878619c5d5"} │
│ 2021-04-30T14:32:11.783Z DEBUG [email protected]/awsxray.go:65 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 00000000000000008c21d3878619c5d5"} ││ 2021-04-30T14:32:11.783Z INFO loggingexporter/logging_exporter.go:332 TracesExporter {"#spans": 6}
That is expected since API_PutTraceSegments which specifies a traceid must be: id - A 64-bit identifier for the segment, unique among segments in the same trace, in 16 hexadecimal digits..
Describe the solution you'd like
It'd provide flexibility to add a new parameter on the awsxray exporter (or maybe provide a processor) that could trim the traceId from 128bit to 64bit.
Describe alternatives you've considered
- a new argument on
awsxrayexporter - a processor one can use before the
awsxrayexporter adapting the incoming data to the limitations ofawsxray
Additional context
The logs copied above are produced using the following setup:
[ user-app -(1)-> otel-side-car ] -(2)-> aws-otel-collector -(3)-> Cloudwatch/XRay
where
-
user-appis some user code producing 64bit traceIds using the Jaeger Java client library. -
(1)is Thrift's jaeger protocol -
otel-side-caris the vanilla https://github.com/open-telemetry/opentelemetry-collector/ version 0.25.0 -
(2)is OTLP-gRPC traces -
aws-otel-collectoris this standalone process v 0.9.0. <-- The log output pasted above is produced in this process
Related to https://github.com/open-telemetry/opentelemetry-collector/issues/3072
We have been working on the X-Ray backend to allow 128 bit trace IDs. Once that work is complete, we will be able to support ingesting 128 bit trace IDs.
Like I mentioned in the other thread, there's some misunderstanding between segment and trace here - X-Ray currently only supports 128-bit trace IDs, and the first 32 bits must be a timestamp (it is segment that is 64). So it's not as simple as trimming down to 64 when exporting we need to wait for some significant backend work to support these 0-padded IDs. We understand the importance of supporting legacy instrumentation so are working on support.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
We are also running into incompatible xray traceId issue as highlighted in this issue in Istio.
For more context here is a high level view of our deployment:
ALB --> Istio (Envoy) --> user-app (OTel Auto Instrumentation) --> OTel Collector --> XRay
- ALB generates TraceId using ALB X-Amzn-Trace-Id header
- Istio does not recognize the ALB TraceId and generates new 128bit traceId and adds x-b3-traceid header
- When we attach OTel Collector to send Istio Envoy traces to XRay , the envoy traceId are not compatible. (It requires some custom massaging logic and does not work out of the box)
- Once traffic reach application layer , auto instrumentation utilizes istio trace id set as part of x-b3-traceid and sends the same to collector. Collector fails to export the same to XRay because of invalid traceId
Workaround with Istio
- Disable Istio Tracing for now by disabling enableTracing: false in MeshConfig
- Once disabled, OTel library will utilize ALB Trace ID or generate a new compatible traceId if no headers ate set.
Long Term Option
- ALB needs to provide propogation of traceId using standard headers: x-b3 headers
- XRay needs to support random traceId rather relying on epoch or there needs to be a translation layer in the library , that can provide valid xray trace id. This will allow us to export Istio Traces to XRay using OTel Collector, and also allow us to leverage OTel auto instrumentation within istio mesh with tracing enabled.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
@Aneurysm9 I'm not sure which milestone to include this under to ensure it doesn't become stale - what do you think?
@willarmiros ADOT Collector Backlog is appropriate for now. We can move it to a more specific milestone as appropriate when more information is available.
X-Ray backend now supports W3C trace id (in X-Ray text format), i.e. the timestamp validation is now removed from both the collector and backend https://aws.amazon.com/about-aws/whats-new/2023/10/aws-x-ray-w3c-format-trace-ids-distributed-tracing/
For integrated AWS services, there is no change for now, the ids are still generated in X-Ray text format and propagated using X-Ray's HTTP headers.
