Example of internal toolchain usage

[com6056]

Describe the bug

I can't seem to get the internal toolchain working, it would be great if there was an example of how to use it.

Reproduction steps

Here's what I have so far in my WORKSPACE:

http_archive(
    name = "rules_erlang",
    sha256 = "dea3fb406b1e00dfbea300553fd3a3678e8079f763ba5d7c808f2c1e8561d9b6",
    strip_prefix = "rules_erlang-3.11.3",
    urls = ["https://github.com/rabbitmq/rules_erlang/archive/refs/tags/3.11.3.zip"],
)

load(
    "@rules_erlang//:rules_erlang.bzl",
    "erlang_config",
    "internal_erlang_from_github_release",
    "rules_erlang_dependencies",
)

rules_erlang_dependencies()

erlang_config(
    internal_erlang_configs = [
        internal_erlang_from_github_release(
            name = "24",
            sha256 = "0b57d49e62958350676e8f32a39008d420dca4bc20f2d7e38c0671ab2ba62f14",
            version = "24.3.4.5",
        ),
    ],
)

load("@erlang_config//:defaults.bzl", "register_defaults")

register_defaults()

When I try to build something though, I can't seem to find the right incantation of platform/toolchain flags to get things working:

bazel build :hello_world --toolchain_resolution_debug="@rules_erlang.*" --platforms @rules_erlang//platforms:erlang_24_platform
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//external:erlang_unknown_unknown_toolchain; mismatching values: erlang_unknown
INFO: ToolchainResolution:   Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: execution @local_config_platform//:host: Selected toolchain @erlang_config//external:erlang_unknown_unknown_toolchain
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//24:erlang_24_3_toolchain; mismatching values: erlang_24
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//24:erlang_24_3_toolchain; mismatching values: erlang_24_3
INFO: ToolchainResolution: Target platform @rules_erlang//platforms:erlang_24_platform: Selected execution platform @local_config_platform//:host, type @rules_erlang//tools:toolchain_type -> toolchain @erlang_config//external:erlang_unknown_unknown_toolchain
INFO: ToolchainResolution: Target platform @rules_erlang//platforms:erlang_24_platform: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//external:erlang_unknown_unknown_toolchain; mismatching values: erlang_unknown
INFO: ToolchainResolution:   Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: execution @local_config_platform//:host: Selected toolchain @erlang_config//external:erlang_unknown_unknown_toolchain
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//24:erlang_24_3_toolchain; mismatching values: erlang_24
INFO: ToolchainResolution:     Type @rules_erlang//tools:toolchain_type: target platform @rules_erlang//platforms:erlang_24_platform: Rejected toolchain @erlang_config//24:erlang_24_3_toolchain; mismatching values: erlang_24_3
INFO: ToolchainResolution: Target platform @rules_erlang//platforms:erlang_24_platform: Selected execution platform @local_config_platform//:host, type @rules_erlang//tools:toolchain_type -> toolchain @erlang_config//external:erlang_unknown_unknown_toolchain
INFO: Analyzed target //:hello_world (0 packages loaded, 439 targets configured).
INFO: Found 1 target...
ERROR: /Users/jrodgers/tmp/build_output/ee5503adaf6f5845e8c6c516e1053150/external/erlang_config/external/BUILD.bazel:12:16: Validating otp at /usr failed: (Exit 127): bash failed: error executing command (from target @erlang_config//external:otp-external) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/bin/bash: line 3: /usr/bin/erl: No such file or directory
Target //:hello_world failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.380s, Critical Path: 0.02s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

Expected behavior

I expect the internal toolchain to work, I'm assuming I'm just doing something wrong though!

Additional context

Tried on both macOS and Linux, no luck.

[com6056]

Managed to get past this by defining my own platforms:

platform(
    name = "erlang_24_platform",
    constraint_values = [
        "@erlang_config//:erlang_24",
    ],
    parents = ["@local_config_platform//:host"],
)

platform(
    name = "erlang_internal_platform",
    constraint_values = [
        "@erlang_config//:erlang_internal",
    ],
    parents = ["@local_config_platform//:host"],
)

And then passing --platforms :erlang_24_platform --host_platform :erlang_internal_platform. Now it seems to be failing to get the toolchain, assuming there's some allow-network flag or tag I need now:

curl: (7) Failed to connect to github.com port 443 after 4 ms: Couldn't connect to server

[com6056]

Ah, had to remove build --sandbox_default_allow_network=false from my .bazelrc, now onto this:

/bin/bash: line 5: sha256sum: command not found

I'm assuming this is due to me running it on macOS and the Homebrew PATH not being passed in.

[HoloRin]

Hi @com6056,

Apologies for the delayed response, I had been away on summer vacation. I'm glad you were able to make some progress on this on your own. I suppose the documentation could be improved a bit, at the very least.

Managed to get past this by defining my own platforms:

For now that is required, as by default we always resolve to "external" erlang by default. This was deemed the least surprising default behavior since most erlang developers already have erlang installed on their system.

I've had a look at the platforms that rules_erlang defines by default in @erlang_config and these could probably be improved so that something like --platforms @erlang_config//:erlang_24_platform --host_platform @erlang_config//:erlang_internal_platform would work out of the box. However if I remember correctly, we tried this some time ago and there were some limitations with bzlmod that kept it from working. Those may be fixed now, so it's probably worth another try.

/bin/bash: line 5: sha256sum: command not found

We primarily use internal erlangs with RBE, and a known docker image. There are therefore some assumptions about what's available on the host that have been overlooked. It seems that we do expect sha256sum is available (https://github.com/rabbitmq/rules_erlang/blob/06c5a814428b60a59df1feb714a9c1f7418e5eea/private/erlang_build.bzl#L72) which is not ideal. The easiest thing would be to make it available by default on your system. We would like to remove as many of those assumptions as we can over time, though getting the erlang_build rule to work properly with zero assumptions about the host is not at the moment a high priority for us. PRs to this effect are welcome. It may be possible to leverage https://github.com/bazelbuild/bazel/blob/master/tools/build_defs/hash/hash.bzl to remove the sha256sum dependency in particular.

[HoloRin]

Oh, and one other thing

erlang_config(
    internal_erlang_configs = [
        internal_erlang_from_github_release(
            name = "24",
            version = "24.3.4.5",
        ),
    ],
)

should skip the sha256sum command, and allow you to check if the rest of the build would succeed

[com6056]

Sorry for the delay! Starting to dig into this again now that I have some time, I'll give your suggestions a try and let you know how it goes 🤞

One thing is that my end goal is to be able to build Elixir apps with this and what's available in https://github.com/rabbitmq/rabbitmq-server/tree/main/bazel/elixir, do you know if it will actually be possible or does it still need a lot of work before it can be used ourside of rabbitmq-server? I saw https://github.com/rabbitmq/rules_erlang/issues/91 but wasn't sure if that was still the current status or not.

Also, I'm a bit confused on the terminology, is internal_erlang_from_github_release hermetic, as in it handles grabbing all the dependencies you need from the release or do you still need system-available dependencies too? Also what is the difference between the internal and external erlang? Is external assuming everything is already installed on the system?

Thanks!

[HoloRin]

internal_erlang_from_github_release isn't hermetic at this point. You are correct that it just assumes the erlang deps are available on the host. The reason that is came about is that rabbitmq-server tests regularly with multiple erlangs in CI, and we used to have a docker image for RBE with each erlang we needed. Now we have a common image with the erlang deps, and bazel builds just erlang so that it's easy to change erlang versions and leave the base image alone. Ideally it would be hermetic, but I don't think that's a trivial task.

Yes, internal means erlang built as part of the bazel build, and external means it's already there on the host.

As for #91, it's probably usable, but I consider it suboptimal since we're just wrapping mix, and at this point I can say that I really don't like mix. I remember running into something recently where I took a look at using elixirc directly. https://github.com/rabbitmq/rabbitmq-server/tree/rin/rules_elixir I made progress, but producing escripts required copying a bit the mix source from elixir and making some private functions public. I think I solved the problem some other way and we moved on, but that branch probably still represents the general approach to be taken for a proper rules_elixir.

[HoloRin]

https://github.com/rabbitmq/rules_elixir/tree/main/examples/internal-elixir serves as an example. Some of the setup is demonstrated in https://github.com/rabbitmq/rules_elixir/blob/42f412bf606eb75a5b090c8b41dbc9e4b3aecd90/.github/workflows/test.yaml#L85-L129