proposal: allow traversing 128-bit IDs via B3

[emirot]

trafficstars

As explained in this original thread, for now all Datadog APM (except java 0.79) are using trace id length of 64 bits. Unfortunately, all my other systems are using a trace length of 128 bits, as it's now the default in istio/envoy and jaeger. As a result all the traces going through that service will lose the original trace. Is this something that could be fixed in this Datadog APM?

[emirot]

To better explain what I mean: Dockerfile

FROM golang:1.16 AS build

ENV CGO_ENABLED=0
WORKDIR /go/src/nolan-golang-datadog
COPY . .

RUN go install -v ./...
RUN go test -v ./...

FROM debian

ENV DD_PROPAGATION_STYLE_INJECT=B3
ENV DD_PROPAGATION_STYLE_EXTRACT=B3

COPY --from=build /go/bin/nolan-golang-datadog /

ENTRYPOINT [ "/nolan-golang-datadog"]

main.go

package main

import (
	"crypto/tls"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"

	"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)


func nolan(w http.ResponseWriter, r *http.Request) {
	span, ctx := tracer.StartSpanFromContext(r.Context(), "post.process")
	defer span.Finish()
	answer := ""
	for name, values := range r.Header {
		for _, value := range values {
			answer += fmt.Sprintf("Header[%s]=(%s)\n", name, value)
		}
	}
	http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
	url := "https:/mycloudfunction.com/pythonheaders"
	req, err := http.NewRequest("GET", url, nil)

	if err != nil {
		log.Fatalf("failed http call:%v\n", err)
	}
	req = req.WithContext(ctx)
	// Inject the span Context in the Request headers
	err = tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header))
	if err != nil {
		// Handle or log injection error
	}
	req.Header.Add("nolan", "test")
	resp, err := http.DefaultClient.Do(req)
	respHeader, err := ioutil.ReadAll(resp.Body)

	fmt.Fprintf(w, answer+"\n"+string(respHeader)+"====END====\n"+fmt.Sprint(span.Context().TraceID())+"\n"+fmt.Sprint(span.Context().SpanID()))
}


func main() {
	tracer.Start()
	defer tracer.Stop()
	http.HandleFunc("/", nolan)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Result:

Header[X-Envoy-Attempt-Count]=(1)
Header[X-Forwarded-Proto]=(https)
Header[X-Forwarded-For]=(...)
Header[User-Agent]=(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36)
Header[Accept]=(text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9)
Header[Sec-Fetch-User]=(?1)
Header[Accept-Encoding]=(gzip, deflate, br)
Header[X-Envoy-External-Address]=(...)
Header[Content-Length]=(0)
Header[X-B3-Spanid]=(91b0bb69dd0ebbe4)
Header[X-Forwarded-Port]=(443)
Header[X-Amzn-Trace-Id]=(Root=1-6080db51-035a132330888d701f4caa25)
Header[Sec-Fetch-Site]=(cross-site)
Header[Sec-Fetch-Mode]=(navigate)
Header[Accept-Language]=(en-US,en;q=0.9)
Header[Upgrade-Insecure-Requests]=(1)
Header[Sec-Fetch-Dest]=(document)
Header[X-B3-Traceid]=(f13206f517ef009ede4e298bb884dba5)
Header[X-B3-Parentspanid]=(de4e298bb884dba5)
Header[Cache-Control]=(max-age=0)
Header[Sec-Ch-Ua]=("Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99")
Header[Sec-Ch-Ua-Mobile]=(?0)
Header[Cookie]=(_ga=GA1.2.507852972.1615311433; zjs_user_id=null; zjs_anonymous_id=%!l(MISSING)ocal-dev-unique-id%!;(MISSING) _gcl_au=1.1.1102910516.1615311434)
Header[X-Request-Id]=(ecb3b302-4fa3-467c-b857-e6924d5c3e11)
Header[X-B3-Sampled]=(0)

Request headers:
Content-Type: text/plain
Host: my.cloudfunctions.net
User-Agent: Go-http-client/2.0
Accept-Encoding: gzip
Forwarded: for="52.24.219.210";proto=https
Function-Execution-Id: nudf82zzmx53
Nolan: test
Traceparent: 00-c2392293b885b34e1c34c2f66afca1e0-27d5debd7a037d4a-00
X-Appengine-City: boardman
X-Appengine-Citylatlong: 45.839855,-119.700583
X-Appengine-Country: US
X-Appengine-Https: on
X-Appengine-Region: or
X-Appengine-Request-Log-Id: 6080db5100ff0925753e9c14df0001737e63363635623534333262383134623938332d7470000136326366383831376331336162363533613765363566353764376237353864353a36000100
X-Appengine-Timeout-Ms: 599999
X-Appengine-User-Ip: ...
X-B3-Sampled: 1
X-B3-Spanid: 411ac017589d905e
X-B3-Traceid: 411ac017589d905e
X-Cloud-Trace-Context: c2392293b885b34e1c34c2f66afca1e0/2870445242895793482
X-Forwarded-For: ....
X-Forwarded-Proto: https
Connection: close

====END====
4691273168363294814
4691273168363294814

As you can see the X-B3-Traceid is different. I guess it's because of https://github.com/DataDog/dd-trace-go/blob/v1/ddtrace/ddtrace.go#L71

[gbbr]

Hi @emirot

I can certainly understand the issue. Thanks for all the detail you have provided.

This isn't strictly a dd-trace-go related problem, it's more of a product wide concern. As it was explained in the other issue, all of Datadog APM uses 64-bit IDs. We are not currently able to easily change that, although, it might be somewhere on the horizon. If you'd like to find out more about this, or when we plan to change to 128-bits, I'd recommend reaching out through our support and they may be able to give you a much higher level view.

At best, we may be able to introduce a change similar to the one on the Java side, where we would preserve a traversing 128-bit. We'll have to look into that a bit.

[gbbr]

I renamed the issue, please let me know if I was inaccurate in any way or misunderstood what you're asking.

[emirot]

@gbbr thank you for updating the issue, yes I already have a ticket open #474435 regarding that.

[katiehockman]

This is something that is planned, and we will update you once it's available to try. Thanks for filing the issue, and for your patience 👍

[katiehockman]

128 bit trace id support is now completed and released. If you experience any issues with this, please file an issue with our support team so we can investigate!