Document setting up a cross-compiling toolchain with cgo

[jayconrod]

Setting up C/C++ cross-compiling toolchains in Bazel is non-trivial. We should document how to do this in a way that works with cgo.

[prestonvanloon]

+1

Specifically looking for Linux to Mac|Windows cross-compilation for projects with cgo.

[prestonvanloon]

@jayconrod is there any support for this documentation in the near future? I've reached a dead end trying to get cgo to produce non-empty archives for cgo libraries when cross compiling for windows.

[jayconrod]

@prestonvanloon Does #1956 solve your problem with non-empty archives? It should be released soon.

About setting up cross-compilation, I wrote Configuring a custom C toolchain some time ago. It doesn't get into cross-compilation specifically, but maybe it will be helpful.

[prestonvanloon]

Not quite. We have this target here https://github.com/prysmaticlabs/bazel-go-ethereum/blob/master/crypto/secp256k1/BUILD.bazel#L4-L50 which happily produces an empty .a file when cross-compiling to windows. (That is, when you set --platforms=@io_bazel_rules_go//go/toolchain:windows_amd64 from a linux system)

Generated file looks something like this:

!<arch>
__.PKGDEF       0           0     0     644     80        `
go object windows amd64 go1.11.5 X:framepointer
----


$$B
i^@^G^@^@^Eempty^A^@^A^@^@^@
$$
_go_.o          0           0     0     644     82        `
go object windows amd64 go1.11.5 X:framepointer
----


!
^@^@go19ld^A^@ÿ^@^@^@^@^@^@ÿÿgo19ld

[excavador]

Any updates?

[rcorre]

I've managed to to cross compile from 64-bit linux to 32-bit Windows. I may have overcomplicated, but it seems to work.

I borrowed a lot from https://docs.bazel.build/versions/3.3.0/tutorial/cc-toolchain-config.html.

The following is tested with;

• bazel 3.0.0
• rules_go cfa1ca3661c76796729c859de5b141298374940a
• mingw-w64-i686 installed on Ubuntu Bionic. Ideally you'd build the compiler yourself to make this hermetic.

1. First, define your cross-compiling toolchain in toolchain/cc_toolchain_config.bzl:

load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES")
load(
   "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl",
   "feature",
   "flag_group",
   "flag_set",
   "tool_path",
)

def _impl_32(ctx):
    # TODO: make hermetic
    tool_paths = [
        tool_path(
            name = "gcc",
            path = "/usr/bin/i686-w64-mingw32-gcc",
        ),
        tool_path(
            name = "ld",
            path = "/usr/bin/i686-w64-mingw32-ld",
        ),
        tool_path(
            name = "ar",
            path = "/usr/bin/i686-w64-mingw32-ar",
        ),
        tool_path(
            name = "cpp",
            path = "/usr/bin/i686-w64-mingw32-g++",
        ),
        tool_path(
            name = "gcov",
            path = "/bin/false",
        ),
        tool_path(
            name = "nm",
            path = "/bin/false",
        ),
        tool_path(
            name = "objdump",
            path = "/bin/false",
        ),
        tool_path(
            name = "strip",
            path = "/bin/false",
        ),
    ]
    features = [
        feature(
            name = "default_linker_flags",
            enabled = True,
            flag_sets = [
                flag_set(
                    actions = [
                        ACTION_NAMES.cpp_link_executable,
                        ACTION_NAMES.cpp_link_dynamic_library,
                        ACTION_NAMES.cpp_link_nodeps_dynamic_library,
                    ],
                    flag_groups = ([
                        flag_group(
                            flags = [
                                "-lstdc++",
                            ],
                        ),
                    ]),
                ),
            ],
        ),
    ]
    return cc_common.create_cc_toolchain_config_info(
        ctx = ctx,
        cxx_builtin_include_directories = [
            "/usr/i686-w64-mingw32/include",
            "/usr/lib/gcc/i686-w64-mingw32"
        ],
        features = features,
        toolchain_identifier = "local",
        host_system_name = "local",
        target_system_name = "windows",
        target_cpu = "@platforms//cpu:x86_32",
        target_libc = "unknown",
        compiler = "mingw",
        abi_version = "unknown",
        abi_libc_version = "unknown",
        tool_paths = tool_paths,
    )

cc_toolchain_config_32 = rule(
    implementation = _impl_32,
    attrs = {},
    provides = [CcToolchainConfigInfo],
)

2. Add a BUILD file for your toolchain at toolchain/BUILD.bazel

package(default_visibility = ["//visibility:public"])

cc_toolchain_suite(
    name = "mingw_suite",
    toolchains = {
        "x86_32": ":mingw_toolchain_32",
        "x86_32|mingw": ":mingw_toolchain_32",
    },
)

filegroup(name = "empty")

cc_toolchain(
    name = "mingw_toolchain_32",
    all_files = ":empty",
    compiler_files = ":empty",
    dwp_files = ":empty",
    linker_files = ":empty",
    objcopy_files = ":empty",
    strip_files = ":empty",
    supports_param_files = 0,
    toolchain_config = ":mingw_toolchain_config_32",
    toolchain_identifier = "mingw-toolchain-32",
)

load(
    ":cc_toolchain_config.bzl",
    "cc_toolchain_config_32",
)

cc_toolchain_config_32(name = "mingw_toolchain_config_32")

toolchain(
    name = "cc-toolchain-mingw",
    exec_compatible_with = [
        "@platforms//cpu:x86_64",
        "@platforms//os:linux",
    ],
    target_compatible_with = [
        "@platforms//cpu:x86_32",
        "@platforms//os:windows",
    ],
    toolchain = ":mingw_toolchain_32",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

3. Define a platform in your main BUILD.bazel that will select this toolchain:

platform(
    name = "windows_386_cgo",
    constraint_values = [
        ":mingw",
    ],
    parents = ["@io_bazel_rules_go//go/toolchain:windows_386_cgo"],
)

4. Register your toolchain in WORKSPACE:

register_toolchains(
    "//toolchain:cc-toolchain-mingw",
)

5. Create a build config in .bazelrc

build:go_win32 --host_crosstool_top=@bazel_tools//tools/cpp:toolchain
build:go_win32 --host_platform=@local_config_platform//:host
build:go_win32 --crosstool_top=//toolchain:mingw_suite
build:go_win32 --cpu=x86_32
build:go_win32 --compiler=mingw
build:go_win32 --platforms=:windows_386_cgo

6. Build

bazel build ${target} --config=go_win32

That's it! You should be able to run this on a 64-bit linux machine (with the 32-bit mingw toolchain installed) and produce a Go binary for 32-bit Windows.

[albertocavalcante]

I know it has been a while but I believe this is much needed.

Note on crosstool.rst removal

About setting up cross-compilation, I wrote Configuring a custom C toolchain some time ago. It doesn't get into cross-compilation specifically, but maybe it will be helpful.

I wanted to point out that crosstool.rst has been recently removed by https://github.com/bazelbuild/rules_go/pull/3986 as it's "the old way in Bazel to define a C/C++ toolchain".

Current documentation at cross_compilation.md

I think the current cross-compilation documentation is located at https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/cross_compilation.md, which says:

By default, cross-compilation will cause Go targets to be built in "pure mode", which disables cgo; cgo files will not be compiled, and C/C++ dependencies will not be compiled or linked.

Cross-compiling cgo code is possible, but not fully supported. You will need to [define and register a C/C++ toolchain and platforms]. You'll need to ensure it works by building cc_binary and cc_library targets with the --platforms command line flag set. Then, to build a mixed Go / C / C++ project, add pure = "off" to your go_binary target and run Bazel with --platforms.

I would like to confirm: by stating "not fully supported", it only means that it is a supported use case although requires additional work other than simply using rules_go, is that interpretation correct?

If so, would be great to have a documentation that explains in detail how to do this, even if it's an opinionated but concrete way with a sample. By only referencing https://bazel.build/extending/toolchains#toolchain-definitions it doesn't help much in my opinion.

If it's relevant, I was thinking in contribute with a guide or sample using https://github.com/uber/hermetic_cc_toolchain but it has the caveat of not supporting OSX at the moment https://github.com/uber/hermetic_cc_toolchain/issues/10.

@prestonvanloon I've seen you did a hack for prysm at https://github.com/prysmaticlabs/prysm/blob/develop/tools/cross-toolchain/darwin_cc_hack.bzl. Even if it's not hermetic, does it work?

I'll also give a try to https://github.com/bazel-contrib/toolchains_llvm, if anyone knows whether it works or not, please let me know.

I'd love to know if anyone has thoughts on this or is aware of any guides, tutorials, samples that would make setting this up easier. Thank you!