auto instrument failed: offset not found: net/http.response:status

[droger88]

Describe the bug

We are using go auto instrumentation with opentelemetry operator, the sidecar container is not able to start due to the following error

{"level":"error","ts":1711045263.4401302,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:175","msg":"error while loading probes, cleaning up","name":"net/http/server","error":"offset not found: net/http.response:status (1.21.8)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:175\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:125\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:152\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"info","ts":1711045263.4401646,"logger":"Instrumentation.Manager.net/http/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/server"}
{"level":"info","ts":1711045263.4401739,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:200","msg":"Cleaning bpffs"}
{"level":"error","ts":1711045263.4401944,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"offset not found: net/http.response:status (1.21.8)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}

Environment

• Go Version: go1.19.3 darwin/arm64
• Instrument version: otel/autoinstrumentation-go:v0.10.1-alpha

To Reproduce

Steps to reproduce the behavior:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-go: "monitoring/instrumentation"
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: '/app/server'
    spec:
      containers:
        - name: app
          image: app-image
          env:
          - name: OTEL_SERVICE_NAME
            value: my-api-dev

Expected behavior

sidecar instrument successfully starts

Additional context

we used to use otel/autoinstrumentation-go:v0.7.0-alpha, and it works.

[mannro]

same here... in v0.8.0-alpha was added this https://github.com/open-telemetry/opentelemetry-go-instrumentation/pull/428

[RonFed]

@droger88 According to the log you attached, the instrumented app is built with go 1.21.8 which is not supported in our latest official release. @MrAlias should we release a newer version?

[MrAlias]

@droger88 According to the log you attached, the instrumented app is built with go 1.21.8 which is not supported in our latest official release. @MrAlias should we release a newer version?

Yes! I can get it out tomorrow.

[DasMagischeToastbrot]

@MrAlias thank you very much, will you also support v1.22?

[MA3CIN]

@MrAlias @RonFed Hi, i'm still running into this issue with both versions 1.22.1 and 1.21.9. According to the README compatibility section (https://github.com/open-telemetry/opentelemetry-go-instrumentation?tab=readme-ov-file#compatibility) both 1.21 and 1.22 GO versions should be supported - does that mean that the minor versions (1.21.9 and so on) are not supported and will not be supported? The 1.21.0 version works like a charm

[MrAlias]

@MrAlias thank you very much, will you also support v1.22?

1.22.0 and 1.22.1 should be supported in the latest release. Please open another issues if you are still having issues.

[MrAlias]

@MrAlias @RonFed Hi, i'm still running into this issue with both versions 1.22.1 and 1.21.9. According to the README compatibility section (https://github.com/open-telemetry/opentelemetry-go-instrumentation?tab=readme-ov-file#compatibility) both 1.21 and 1.22 GO versions should be supported - does that mean that the minor versions (1.21.9 and so on) are not supported and will not be supported? The 1.21.0 version works like a charm

@MA3CIN This was released today. We'll need some time to work on support. I imagine a few days we'll have something.

[MA3CIN]

Got it, thanks for the quick response!

[ahilmathew]

Got a similar error -

{"level":"error","ts":1715663116.6580272,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:180","msg":"error while loading probes, cleaning up","name":"net/http/server","error":"offset not found: net/http.response:status (1.22.3)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:180\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:127\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:151\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":1715663117.146594,"logger":"Instrumentation.Manager.database/sql/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"database/sql/client"}
{"level":"info","ts":1715663117.4265897,"logger":"Instrumentation.Manager.google.golang.org/grpc/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/client"}
{"level":"info","ts":1715663117.716724,"logger":"Instrumentation.Manager.google.golang.org/grpc/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"google.golang.org/grpc/server"}
{"level":"info","ts":1715663117.7167873,"logger":"Instrumentation.Manager.net/http/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/server"}
{"level":"info","ts":1715663117.7167969,"logger":"Instrumentation.Manager.net/http/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/client"}
{"level":"info","ts":1715663117.7168033,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:205","msg":"Cleaning bpffs"}
{"level":"error","ts":1715663117.7170658,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"offset not found: net/http.response:status (1.22.3)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

Any insights on what is causing this?

[MA3CIN]

@ahilmathew Offset not found - most likely incompatible versions of the Otel operator and Go version of the service you're trying to instrument. Which versions are you using?

[ahilmathew]

Thanks @MA3CIN. I had a look now. We were on 1.22.3 which isn’t supported yet I think. Switching to go version 1.22.1 worked.

[MrAlias]

We plan to release support for Go 1.22.3 (and 1.21.10) in a few days.

[svithalani-washpost]

Did you release support for 1.22.3 ? I am getting the same error