Better API for cross-compilation support

[benradf]

Is your feature request related to a problem? Please describe. In principle cross-compilation is already supported thanks to the nixpkgs system and targetSystem parameters. However this requires the user to write their own nix expressions and does not integrate nicely with how Bazel handles cross-compilation. Hence the need for a better API around this.

Describe the solution you'd like We can generate toolchain instances for a set of exec and target platform pairs and apply appropriate Bazel constraints automatically. This needs to be configurable. The nodejs toolchain is an example where this is somewhat supported.

For packages, we want to generate instances for a set of platforms. Build tool packages may incur separate exec and target constraints (i.e. a package with a binary that runs during the build and a shared library for the target platform). There should be an alias target that uses select to pick the appropriate package instance.

[layus]

Just a quick note here: Aliases are cumbersome as you can only alias targets, not full-blown external repositories. So you need to know all the bazel targets declared in a nixpkgs_package to alias them.

[aherrmann]

@layus Yes, that's true - and annoying. Unfortunately, I'm not sure what a better approach may look like. Short of defining toolchains.

[DolceTriade]

I'm interesting in tackling this. I put together a hacky proof of concept of a pretty involved repository set up that works off of https://github.com/tweag/rules_nixpkgs/pull/240 where I can cross compile for linux x86_64 on my M1 macbook, and then run the resulting binaries in Docker using --run_under.

There are only minor changes required to rules_nixpkgs (namely, that the toolchain CPU for C++ is hard coded to the system CPU).

I ended up with something like this in my WORKSPACE.bazel

nixpkgs_cc_configure(
    name = "nixpkgs_crossosx_cc",
    cross_cpu = "k8",
    exec_constraints = ["@platforms//os:osx", "@platforms//cpu:arm64"],
    nix_file = "//bazel/nix/cc:osxcross_cc.nix",
    nixopts = [
        "--arg",
        "ccPkgs",
        "import <nixpkgs> { crossSystem = \"x86_64-linux\";}",
        "--show-trace",
    ],
    repository = "@nixpkgs",
    target_constraints = [
        "@platforms//cpu:x86_64",
        "@platforms//os:linux",
        "@rules_nixpkgs_core//constraints:support_nix",
    ],
)

and in my bazelrc

build:osxcross --cpu=k8 --crosstool_top=@nixpkgs_crossosx_cc//:toolchain --platforms=//bazel/nix/cc:linux_x86_64 --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/platforms:host --host_crosstool_top=@nixpkgs_config_cc//:toolchain
run:osxcross --run_under="//.bazel-lib/tools:run_under_docker.sh"
test:osxcross --run_under="//.bazel-lib/tools:run_under_docker.sh"

For Linux only dependencies, I ended up writing a macro that looks like:

def linux_nixpkgs_package(name, repository, build_file = None):
    nixpkgs_package(
        name = name,
        repository = repository,
        attribute_path = name,
        build_file = build_file,
        nix_file_content = """import <nixpkgs> { config = {}; overlays = []; system = "x86_64-linux"; }""",
    )

So I could define something like:

    linux_nixpkgs_package(
        name = "libnfnetlink",
        repository = "@nixpkgs",
        build_file = "//third_party/nix:BUILD.libnfnetlink",
    )

I we could write macros that could automatically handle the select() boiler plate as well if we want to have nixpkgs_package dependencies that depend on a config_setting or something?

That said, initially I'd like to maybe take over #240 and fix the hardcoded CPU in toolchains/cc/cc.bzl. Then we can improve the API around cross compiled nixpackage selection...as technically users can easily create their own macros for this too.

There's probably a lot of cleanup that's viable in my set up, but I think this is a good start. WDYT?

[benradf]

Hi @DolceTriade, cross compilation is something that comes up a lot since the M1 was released, so it's great that you're interested in tackling this issue.

That said, initially I'd like to maybe take over #240 and fix the hardcoded CPU in toolchains/cc/cc.bzl. Then we can improve the API around cross compiled nixpackage selection...as technically users can easily create their own macros for this too.

I think this would be a good approach. Getting #240 merged has been on my list of things to do for a while — there were some conflicts but I just resolved them so I think it should be okay now. Feel free to open a PR based on that branch to fix the hardcoded CPU in toolchains/cc/cc.bzl.

If you feel like it, I think it would be great to add your proof of concept as an example here. Then we have something concrete to refer people to when they want to do cross compilation, even before we undertake the more lengthy task of improving the API.

[benradf]

@DolceTriade #240 was just merged, so you can base your PR to fix the hardcoded CPU off master now.

[jcpetruzza]

Thanks for adding this! I'm currently trying this out for the Linux -> MacOS direction (both x86-64); one blocker I notice is that the cross-toolchain targeting darwin wants to to use the gold linker, which only works on ELF.

As far as I can see, the problem is that the generated cc_toolchain_config() looks like this:

cc_toolchain_config(
    name = "local",
    cpu = "darwin",
    ...
    link_flags =  ["-fuse-ld=gold",
      ...    
    ],
   ...
)

which I think comes from the nix expression that generates the toolchain info, since it add that to the link-flags whenever ld.gold is available. I guess an extra needs need to be added that takes into account the target?