Golang frames are not creating executable filenames in mappings

[rockdaboot]

Golang frames are treated as interpreter frames and thus in the mappings there the filename of the executable/binary isn't stored.

While this makes sense for real interpreted languages, we should send the filename for Go frames.

Code is around here: https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/6f2a54527e19e65e266a6c89b9c1415d48048b47/reporter/internal/pdata/generate.go#L156

[florianl]

The process executable name is always set - independent of the frame type:

https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/20153416e82fc8ea4cc6016f1c72f7fa3cce473b/reporter/internal/pdata/generate.go#L256-L257

For interpreted frames, including Go, the build ID of the executable, that is required for post processing, is not set. But as post processing for symbolization for Go is no longer required, this should be optional going forward.

Update: As Go is handled as interpreted language in the symbolization process, no Mappings are reported for post processing but complete symbolized Function information.

[rockdaboot]

The attribute process.executable.name would work only for fully static binaries. But Go binaries sometimes dynamically load parts of glibc (libc.so.6) - so if the backend wants to symbolize a frame, it needs to know the name libc.so.6 and the hash and not the executable name of the Go binary myAwesomeGoApp.

[florianl]

If a Go process uses cgo to call out to dynamic loaded parts, like libc.so.6, then these specific frames are handled as native frames. And for native frames, the mapping information is reported.

[rockdaboot]

If a Go process uses cgo to call out to dynamic loaded parts, like libc.so.6, then these specific frames are handled as native frames. And for native frames, the mapping information is reported.

Yes, you are right.

So for the backend, we either have to set the filename for Go frames and the hash of the binary - or we can fall back to the process.executable.name but wouldn't have a hash for the binary (which possible isn't bad, as no symbolization step is required for Go frames).

[rockdaboot]

Unknown territory for me: What if a Go executable is dynamically loading shared libraries built with Go? Wouldn't we need the binary filename per frame?

[florianl]

Unknown territory for me: What if a Go executable is dynamically loading shared libraries built with Go? Wouldn't we need the binary filename per frame?

No, we don't need the Go executable to post-process things in this case. Here is a snippet for such a case:

This screenshot is from running code, base don https://gist.github.com/florianl/b6d287c0e4b59372e2652eba1e08c8b7. In the screenshot you can see, that we use cgo to load a Go plugin, end up with a Go frame in the plugin before calling Kernel functions.

[rockdaboot]

That's a good example. Also thanks for sharing the code, looks straight forward.

What we don't see in the flamegraph is the Fib() function. Anyway, what if the main binary loads two different plugins (A and B), both having a function Fib(). Now let's assume that Fib() is CPU intensive and appears in the flamegraph. How are you going to differentiate how much CPU was spent in each of the two different Fib() functions if no executable.file.name is assigned to the frames?

I think we have to provide executable.file.name for each Go frame to be able to assign frames/functions correctly to an executable name. The fallback to use the process.executable.name attribute isn't sufficient.

[christos68k]

Isn't the Filename in Function enough? Expectation being that it'd be different across the two Fib functions.

[rockdaboot]

Isn't the Filename in Function enough? Expectation being that it'd be different across the two Fib functions.

Yes, if it contains the full path like github.com/org/project/app/pluginA/fib. Not sure if this is always the case. Still, the executable file name would be app, even if the frame is inside pluginA.so. That might be confusing.