dd-trace-go
dd-trace-go copied to clipboard
DataDog
[BUG]: Invalid Module Imports
Tracer Version(s)
2.3.1
Go Version(s)
go1.25.3 darwin arm64
Bug Report
ddtrace is depending on datadog-agent, which depends on the go.opentelemetry.io/collector/pdata/internal/data module, but this module was removed from the OTEL project. This leads to a ton of errors when running go mod tidy and causes builds to fail. I found that unless I downgrade to v 1.74.6 this tends to happen for most projects.
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1development
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1development
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1
go: github.com/justworkshr/communications/cmd imports
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer imports
github.com/DataDog/datadog-agent/pkg/trace/stats imports
github.com/DataDog/datadog-agent/pkg/trace/transform imports
github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
go.opentelemetry.io/collector/processor/processortest imports
go.opentelemetry.io/collector/consumer/consumertest imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data: module go.opentelemetry.io/collector/pdata@latest found (v1.46.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data
go: github.com/justworkshr/communications/cmd imports
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer imports
github.com/DataDog/datadog-agent/pkg/trace/stats imports
github.com/DataDog/datadog-agent/pkg/trace/transform imports
github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
go.opentelemetry.io/collector/processor/processortest imports
go.opentelemetry.io/collector/consumer/consumertest imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1development: module go.opentelemetry.io/collector/pdata@latest found (v1.46.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1development
go: github.com/justworkshr/communications/cmd imports
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer imports
github.com/DataDog/datadog-agent/pkg/trace/stats imports
github.com/DataDog/datadog-agent/pkg/trace/transform imports
github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
go.opentelemetry.io/collector/processor/processortest imports
go.opentelemetry.io/collector/consumer/consumertest imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1: module go.opentelemetry.io/collector/pdata@latest found (v1.46.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1
go: github.com/justworkshr/communications/cmd imports
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer imports
github.com/DataDog/datadog-agent/pkg/trace/stats imports
github.com/DataDog/datadog-agent/pkg/trace/transform imports
github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
go.opentelemetry.io/collector/processor/processortest imports
go.opentelemetry.io/collector/consumer/consumertest imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1development: module go.opentelemetry.io/collector/pdata@latest found (v1.46.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1development
Reproduction Code
No response
Error Logs
No response
Go Env Output
AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/jordanblacker/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/jordanblacker/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/2t/mgrzy4694ssb_27bpm3wjmhw0000gp/T/go-build4200598242=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/jordanblacker/justworks/communications/go.mod'
GOMODCACHE='/Users/jordanblacker/go/pkg/mod'
GONOPROXY='github.com/justworkshr/*'
GONOSUMDB='github.com/justworkshr/*'
GOOS='darwin'
GOPATH='/Users/jordanblacker/go'
GOPRIVATE='github.com/justworkshr/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/jordanblacker/go/pkg/mod/golang.org/[email protected]'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/jordanblacker/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/jordanblacker/go/pkg/mod/golang.org/[email protected]/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK=''
PKG_CONFIG='pkg-config'
Hi @jblacker, we'll take a look between today and tomorrow. Are you instrumenting with dd-trace-go API or using our OTel trace provider? Just to make sure our repro code is the most accurate possible.
We're having the same issue. using gopkg.in/DataDog/dd-trace-go.v1 v1.74.7 without otel
@zekth @jblacker We tried to reproduce it but it works for us right now. We created a simple server:
package main
import (
"net/http"
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
)
func main() {
tracer.Start()
defer tracer.Stop()
mux := httptrace.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello World!\n"))
})
http.ListenAndServe(":8080", mux)
}
And we ensured our Go toolchain pulled the exact version:
export GOPROXY=proxy.golang.org,direct
go clean -modcache
go get gopkg.in/DataDog/[email protected]
go mod tidy
go build
Edit: updated example to explicitly import the tracer.
@darccio seems like your example is not importing the tracer
see the example provided:
go: github.com/justworkshr/communications/cmd imports
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer imports
github.com/DataDog/datadog-agent/pkg/trace/stats imports
github.com/DataDog/datadog-agent/pkg/trace/transform imports
github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
go.opentelemetry.io/collector/processor/processortest imports
go.opentelemetry.io/collector/consumer/consumertest imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data: module go.opentelemetry.io/collector/pdata@latest
I have the same issue with the tracer import
@zekth It shouldn't matter for the module resolution. I updated the example and tried. Same non-reproducible result.
This is my go env output:
AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE='auto'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/dario.castane/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/dario.castane/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/jz/200n23xd7fd94f51mt3787qm0000gn/T/go-build3692135858=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/dario.castane/Code/puppies/go-gopkgin/go.mod'
GOMODCACHE='/Users/dario.castane/go/pkg/mod'
GONOPROXY='github.com/DataDog,github.com/darccio/knobs'
GONOSUMDB='github.com/DataDog,go.ddbuild.io'
GOOS='darwin'
GOPATH='/Users/dario.castane/go'
GOPRIVATE='github.com/DataDog,github.com/darccio/knobs'
GOPROXY='proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.25.4/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/Users/dario.castane/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.25.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.4'
GOWORK=''
PKG_CONFIG='pkg-config'
This looks like a repetition of #3059. Probably the workaround should be bumping the right package to the right version.
interestingly the example you shared works on my machine but a project using it doesn't.
go env
AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/xxx/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/xxx/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/c5/v5pjf9md619csnt4hvzyj9jm0000gn/T/go-build942000903=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/xxx/kong/xxx/go.mod'
GOMODCACHE='/Users/xxx/go/pkg/mod'
GONOPROXY='github.com'
GONOSUMDB='github.com'
GOOS='darwin'
GOPATH='/Users/xxx/go'
GOPRIVATE='github.com'
GOPROXY='proxy.golang.org,direct'
GOROOT='/Users/xxxx/go/pkg/mod/golang.org/[email protected]'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/xxx/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/xxx/go/pkg/mod/golang.org/[email protected]/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'
~/kong/xxx feat/redisfx !2 ❯ go mod tidy 19:21:38
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1development
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1development
go: finding module for package go.opentelemetry.io/collector/pdata/internal/data
go: github.com/kong-konnect/xxx/internal/xxx/xxx imports
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer tested by
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.test imports
gopkg.in/DataDog/dd-trace-go.v1/internal imports
go.opentelemetry.io/collector/pdata/pprofile imports
go.opentelemetry.io/collector/pdata/internal/data: module go.opentelemetry.io/collector/pdata@latest found (v1.46.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data
@zekth I meant the imports used in your code. Specific contribs, additional products, etc.
here's the whole indirection from DD:
github.com/DataDog/appsec-internal-go v1.13.0 // indirect
github.com/DataDog/datadog-agent/comp/core/tagger/origindetection v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/opentelemetry-mapping-go/otlp/attributes v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/proto v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.74.0-devel // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/log v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/scrubber v0.72.2 // indirect
github.com/DataDog/datadog-agent/pkg/version v0.72.2 // indirect
github.com/DataDog/datadog-go/v5 v5.8.1 // indirect
github.com/DataDog/dd-trace-go/contrib/99designs/gqlgen/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/IBM/sarama/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/Shopify/sarama/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/aws/aws-sdk-go-v2/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/aws/aws-sdk-go/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/cloud.google.com/go/pubsub.v1/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/confluentinc/confluent-kafka-go/kafka.v2/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/confluentinc/confluent-kafka-go/kafka/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/database/sql/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/elastic/go-elasticsearch.v6/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gin-gonic/gin/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go-chi/chi.v5/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go-chi/chi/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go-redis/redis.v7/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go-redis/redis.v8/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go-redis/redis/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/go.mongodb.org/mongo-driver/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gocql/gocql/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gofiber/fiber.v2/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gomodule/redigo/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/google.golang.org/grpc/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gorilla/mux/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/gorm.io/gorm.v1/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/graph-gophers/graphql-go/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/graphql-go/graphql/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/hashicorp/vault/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/jackc/pgx.v5/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/julienschmidt/httprouter/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/k8s.io/client-go/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/labstack/echo.v4/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/log/slog/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/net/http/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/redis/go-redis.v9/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/redis/rueidis/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/segmentio/kafka-go/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/sirupsen/logrus/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/twitchtv/twirp/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/contrib/valkey-io/valkey-go/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/instrumentation/testutils/grpc/v2 v2.4.0 // indirect
github.com/DataDog/dd-trace-go/v2 v2.4.0 // indirect
github.com/DataDog/go-libddwaf/v4 v4.7.0 // indirect
github.com/DataDog/go-runtime-metrics-internal v0.0.4-0.20250806100345-ca5e7fdaf7b6 // indirect
github.com/DataDog/go-sqllexer v0.1.10 // indirect
github.com/DataDog/go-tuf v1.1.1-0.5.2 // indirect
github.com/DataDog/gostackparse v0.7.0 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/attributes v0.33.0 // indirect
github.com/DataDog/sketches-go v1.4.7 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/collector/component v1.46.0 // indirect
go.opentelemetry.io/collector/featuregate v1.46.0 // indirect
go.opentelemetry.io/collector/internal/telemetry v0.140.0 // indirect
go.opentelemetry.io/collector/pdata v1.46.0 // indirect
go.opentelemetry.io/collector/pdata/pprofile v0.136.0 // indirect
go.opentelemetry.io/collector/semconv v0.128.0 // indirect
go.opentelemetry.io/contrib/bridges/otelzap v0.13.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.38.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/log v0.14.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/sdk v1.38.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
@zekth @jblacker Let me know if the issue is fixed by pulling this dd-trace-go v2 version:
go get github.com/DataDog/dd-trace-go/[email protected]/ktlo/pdata-failure-202511
Wow ya'll were way faster than I expected on this! We were able to resolve the issue using this solution since I didn't see my notifications here:
replace gopkg.in/DataDog/dd-trace-go.v1 => gopkg.in/DataDog/dd-trace-go.v1 v1.74.6
replace github.com/DataDog/datadog-agent/pkg/template => github.com/DataDog/datadog-agent v0.65.1
replace go.opentelemetry.io/collector/pdata => go.opentelemetry.io/collector/pdata v1.44.0
I'll test out the included rev (commenting out these replace calls) and report back shortly.
Are you instrumenting with dd-trace-go API or using our OTel trace provider?
We're instrumenting with the dd-trace-go API. We're not using OTel directly at all.
Unfortunately that branch does not fix the issue due to a conflict it causes in the contrib/net/http/v2 module. I was able to successfully run go mod tidy, but then the application fails to build:
# github.com/DataDog/dd-trace-go/contrib/net/http/v2/internal/wrap
../../go/pkg/mod/github.com/!data!dog/dd-trace-go/contrib/net/http/[email protected]/internal/wrap/roundtrip.go:152:13: assignment mismatch: 1 variable but httpsec.ProtectRoundTrip returns 2 values
../../go/pkg/mod/github.com/!data!dog/dd-trace-go/contrib/net/http/[email protected]/internal/wrap/roundtrip.go:152:43: cannot use req.URL.String() (value of type string) as *http.Request value in argument to httpsec.ProtectRoundTrip
By the way I was not able to run go get the way you posted since it threw "invalid: disallowed version string"; so I went to your branch and was able to successfully pull it by the commit SHA:
go get github.com/DataDog/dd-trace-go/v2@f68b737
@jblacker Thanks for the feedback. That's curious, I tested that go get worked referring to my branch. In any case, it seems that merging this PR and releasing should offer a stable workaround. We'll proceed with that.
We are in the middle of a release freeze, but I'll ask for an exception. In the meantime, if anybody finds this issue, the following go get commands should be considered the best workaround. You can fix the conflict with contrib/net/http/v2 by pulling that contrib from the same commit SHA:
go get github.com/DataDog/dd-trace-go/v2@f68b737
go get github.com/DataDog/dd-trace-go/contrib/net/http/v2@f68b737
go mod tidy
v2.5.0-rc.1 is available for working around the issue. It includes the fix I implemented.
go get github.com/DataDog/dd-trace-go/[email protected]
go get github.com/DataDog/dd-trace-go/contrib/net/http/[email protected]
go mod tidy