Documentation to address errors like "/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found"

[gonzojive]

I'm running into the error in the subject line when building locally and deploying in a docker container.

I can't tell if I need to do anything special to make rules_go build hermetically. Based on the output below, I'm guessing my host system's GLIBC is somehow affecting the binaries being compiled, but I'm not sure.

I thought rules_go and bazel mostly ignored the host system's installed libraries and toolchains. Maybe that is not the default behavior?

Relevant links:

1. Examples on hermetic x86 cc_toolchains? on bazel-discuss
2. https://github.com/bazelbuild/rules_go/issues/1642

GLIBC links:

1. https://github.com/bazelbuild/bazel/blob/master/tools/cpp/unix_cc_configure.bzl
2. https://bazel.build/tutorials/cc-toolchain-config
3. https://docs.bazel.build/versions/5.1.1/skylark/lib/cc_common.html#create_cc_toolchain_config_info documents target_libc with an example value of "glibc-2.2.2"
4. https://sourcegraph.com/search?q=context:global+target_libc&patternType=literal - tensorflow seems to define their own toolchains?
5. https://github.com/bazelRio/bazelRio/blob/1aad71917e2b4e372e295c015b82a8a79c721d2a/bazelrio/toolchains/roborio/toolchain_config.bzl

What version of rules_go are you using?

v0.32.0

What version of gazelle are you using?

v0.25.0

What version of Bazel are you using?

7.1.1

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux 22.04

Any other potentially useful information about your toolchain?

Local execution but running the cgo binary I'm building on Kubernetes using this rules_docker rule:

container_pull(
    name = "golang_container",
    registry = "index.docker.io",
    repository = "golang",
    tag = "1.18.2-bullseye",
)

What did you do?

I used rules_k8s to deploy a program to a cluster using a locally built docker image. I can provide more details later on. The project I'm working in is a lot like this one, except it uses cgo: https://github.com/gonzojive/beam-go-k8s

load("@io_bazel_rules_docker//go:image.bzl", "go_image")

go_image(
    name = "my-image",
    args = [],
    #base = "@distroless//base:base_root_amd64_debian11",
    #base = "@distroless//cc:cc_root_amd64_debian11",
    #base = "@official_ubuntu//image",
    base = "@golang_container//image",
    binary = ":my-program",
    goarch = "amd64",
    goos = "linux",
)

my-image uses cgo

(I can provide more information if needed)

What did you expect to see?

Successful program execution.

What did you see instead?

Error when running the program in the cluster:

... /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /app/data/...)
... /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /app/data/...)

[achew22]

I believe the issue here is that you don't have a statically linked binary. We already have documentation on how to do that here. What additional docs would you be interested in there being?

[gonzojive]

Thanks for the link.

I suppose what I'm looking for is a guide to a reproducible build for code that involves Go with or without C/C++ dependencies.

1. I can't tell if the host system changes the build output or not. e.g., is GLIBC being influenced by my machine?, is the host gcc or go toolchain ever used?

2. Is there anything that can be put into WORKSPACE files to make a build reproducible?

Just some information would be helpful for me, but I figure it will be more useful for the community if that information makes it into the documentation.

On Wed, Jun 1, 2022 at 12:18 PM Andrew Z Allen @.***> wrote:

I believe the issue here is that you don't have a statically linked binary. We already have documentation on how to do that here https://github.com/bazelbuild/rules_go/blob/ba7bdfd6b5118d135ae3f7984c94103510bf4167/go/modes.rst#building-static-binaries. What additional docs would you be interested in there being?

— Reply to this email directly, view it on GitHub https://github.com/bazelbuild/rules_go/issues/3167#issuecomment-1144039589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAUO53SZWPR7TM632HQMNTVM6ZSBANCNFSM5XQH4N5Q . You are receiving this because you authored the thread.Message ID: @.***>

[achew22]

I suppose what I'm looking for is a guide to a reproducible build for code that involves Go with or without C/C++ dependencies.

Why do you believe the builds are not reproducible? If you bazel build twice with a bazel clean in the middle, are you getting different binaries out when you sha256sum the binary? Your build should be deterministic and give the same result between multiple builds, even on different machines. There are no flags that enable reproducibility in any configuration. It should always build reproducibly, there is no way to opt out of that.

[gonzojive]

Please take a look at https://github.com/gonzojive/rules-go-cgo-glibc.

On ubuntu 18.04, the output is

sha256sum bazel-out/k8-fastbuild/bin/program_/program:
d008634d2a4cee180ce1ab47a0ad5274964e76631590294ac72297746a95f5ff  bazel-out/k8-fastbuild/bin/program_/program
===========================
ldd --verbose bazel-out/k8-fastbuild/bin/program_/program:
        linux-vdso.so.1 (0x00007ffe94923000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2d203bb000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2d2019c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2d1fdab000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2d20759000)

        Version information:
        bazel-out/k8-fastbuild/bin/program_/program:
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

On 22.04:

sha256sum bazel-out/k8-fastbuild/bin/program_/program:
0963b7459542358503fb497962cd3bb28c02fb65c86c2324a22e52d9737afe3a  bazel-out/k8-fastbuild/bin/program_/program
===========================
ldd --verbose bazel-out/k8-fastbuild/bin/program_/program:
        linux-vdso.so.1 (0x00007ffeaed85000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8b5dd53000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8b5db2b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8b5de3f000)

        Version information:
        bazel-out/k8-fastbuild/bin/program_/program:
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.32) => /lib/x86_64-linux-gnu/libc.so.6
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

[gonzojive]

I'm guessing there's a way to set it up so that a WORKSPACE-supplied gcc is used to make the build reproducible. It seems pretty complicated to accomplish that, however.

[fmeum]

If you use CGo, both vanilla Go and rules_go will pick up local libraries - without further setup, there simply isn't anything else to use to satisfy the libc requirement. If you want to be sure that you get a completely hermetic build with CGo, you have to:

1. Disable Bazel's auto-detection of the host C/C++ toolchain by adding a line like this to your .bazelrc:

common --repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1

2. Set up a hermetic toolchain such as https://github.com/grailbio/bazel-toolchain in your WORKSPACE.
3. Additionally, use a sysroot (see https://github.com/grailbio/bazel-toolchain/blob/f14a8a5de8f7e98a011a52163d4855572c07a1a3/tests/WORKSPACE#L60-L87 for how to do this with bazel-toolchain).

[gonzojive]

Thanks so much - great information. Consider adding this answer to the official docs of rules_go (and probably rules_cc as well).