go_binary: consistent ELF_NT_GNU_BUILD_ID

[ali5h]

This field is needed for creating RPM debug pckgs. If I build in pure mode non of the binaries have build id set and I have to set them manually to a fixed value using gc_linkopts = ["-B", "0x12345678"]. But in non-pure mode some of the binaries has build id, and some don't. It would be great to have a consistent way of setting this, for example if --stamp is used.

[jayconrod]

in non-pure mode some of the binaries has build id, and some don't.

That's surprising. We shouldn't be setting that in go_binary unless it's done explicitly in gc_linkopts. Is there an example of this you could point us to? I wonder if this is a flag that passed up through some cgo code?

It would be great to have a consistent way of setting this, for example if --stamp is used.

Unfortunately, --stamp is not exposed to Bazel rules implemented in Skylark, so we can't use this. There are some other flags like --define we might be able to use though.

Could you tell us more about your use case? There are a number of different ways we could do this:

• If this value should always be the same for a given target, gc_linkopts is the best way to go. Doesn't seem like this makes sense though.
• If you need to set this to an explicit value determined outside of Bazel, maybe we can pipe something in from --define or --workspace_status_command (which is how we handle regular Go link stamping).
• If this should be an automatically generated hash value for each target, we could generate something in the link action.

[ali5h]

I haven't figure out yet why some of the binaries have this without setting any option, but i know in pure more none of them has it.

My use case is building an rpm. We have and old pipeline which is not fully converted to bazel yet, which packages binaries using rpmbuild. it needs a unique build id per binary. right now we are setting a random value, but something like md5 hash is a better option. it would be great if we can combine md5 hash with git revision to be able to trace back the commit.

I have access to git revision using --workspace_status_command, but gc_linkopts does not support using variables, and it is not per binary unique either.

[jayconrod]

If you need a unique id per binary, it sounds like auto-generating something may be the best option.

I'm told that "go build" in Go 1.10 has started doing this for build / test caching. I'll look into what they're doing.

[jayconrod]

Looked into this a bit more. I don't think the Go internal linker ever sets the build id unless the -B option is used explicitly. However, the external linker (i.e., the C linker) is configured on some systems to set the build id on every link unless explicitly told not to. So I'm guessing you're seeing the build-id set for binaries that were linked with the external linker (i.e., anything with cgo).

If the build id automatically generated by the external linker is good enough, you can force it on in binaries by adding -linkmode=external to gc_linkopts.

go_binary(
    name = "hello",
    srcs = ["hello.go"],
    gc_linkopts = ["-linkmode=external"],
)

Does that work for you?

[ali5h]

interesting, this does not work in pure mode I assume, which is what we are using right now.

[ali5h]

I tried for one binary. Build failed in pure mode and no build id was set in non-pure mode either.

[jayconrod]

Sorry that didn't work. It sounds like a build id needs to be explicitly passed to the internal linker, and it probably should be generated from inputs.

The easiest place for us to implement this would probably be in our link helper, go/tools/builders/link.go. That has access to all the .a files, so if it were passed some extra flag, it could hash those files and pass a -B flag with the hash to the Go linker. WDYT?

How does this work in your old pipeline?

[ali5h]

that makes sense, how the c linker generates the hash? the old pipeline is using go install, and we generated random number as build id!!

[AustinSchuh]

@jayconrod

Stamp is exposed! You can get to it via:

    name = "stamp",
    values = { 
        "stamp": "yes",
    },  
)```

It is only available in macros, but that's a solvable problem by wrapping your rule in a macro.  I've got a macro saying: 

```def convert_stamp(stamp):
  """Converts the provided stamp flag from the rule to a stamp or not bool."""
  if stamp == -1: 
    return select({'@//tools:stamp': True, '//conditions:default': False})
  elif stamp == 0:
    return False
  else:
    return True

[sluongng]

So recently Gitlab folks found that without GNU_BUILD_ID set in the binary, operators/SRE will have a really hard time troubleshooting production problems using perf (link). The Perf data file will default go_binary to have an all-zeros build ID, together with other binaries thus record data are all mixed up.

Plus if you did a new deployment of a new binary version and analyze the old perf.data, the events analysis is completely wrong because perf rely on the build ID to differentiate the binaries.

---

It looks like this issue is quite dated. Ever since this was created, both Bazel and rules_go have been updated quite a lot. We are now relying on x_defs attribute of the starlark rule to provide stamp values when invoking go tool linker: https://github.com/bazelbuild/rules_go/blob/0c8636d7f6341c97fa198cb7e4c0d7a5cc07fcbc/go/private/actions/link.bzl#L138-L149

However, we are setting a hard coded Go BuildID https://github.com/bazelbuild/rules_go/blob/0c8636d7f6341c97fa198cb7e4c0d7a5cc07fcbc/go/private/actions/link.bzl#L157-L161 and we are not using go tool link -B flag which supposed to help us set the GNU Build ID:

go tool link -h
usage: link [options] main.o
  -B note
        add an ELF NT_GNU_BUILD_ID note when using ELF
...

---

So to make these work, I would suggest that we put forward some hardcode keys for (1) Go Build ID and (2) GNU Build ID and allow people to set values of these keys through x_defs stamping mechanism of rules_go.