clodl icon indicating copy to clipboard operation
clodl copied to clipboard
verified publisher icon tweag

Migrate to bzlmod

Open hofbi opened this issue 1 year ago • 15 comments

Migrate the main project to bzlmod and rename all BUILD files to BUILD.bazel

Closes https://github.com/tweag/clodl/issues/53 Closes https://github.com/tweag/clodl/issues/54

hofbi avatar Jun 13 '24 14:06 hofbi

CI probably fails due to some older version of bazel which is used in the nixpkgs, but here I might need your input on which version of packages you want to use.

hofbi avatar Jun 14 '24 07:06 hofbi

CI probably fails due to some older version of bazel which is used in the nixpkgs, but here I might need your input on which version of packages you want to use.

Thank you, @hofbi. That looks good! I'll update nixpkgs and push my changes in a moment...

avdv avatar Jun 14 '24 07:06 avdv

Sounds good, thanks.

What do you guys think about using bazelisk to include bazel? This way you can easily control it from the version file and it is the recommended way for installing bazel. But definitely a follow up PR in case we wanna do that.

Another task would also be to convert the tests to bzlmod.

hofbi avatar Jun 14 '24 08:06 hofbi

The tests include some Nix components and might depend on some of the patches made to Bazel in nixpkgs.

aherrmann avatar Jun 14 '24 09:06 aherrmann

The tests include some Nix components and might depend on some of the patches made to Bazel in nixpkgs.

Should we also migrate the tests to bzlmod in this PR?

hofbi avatar Jun 14 '24 10:06 hofbi

Should we also migrate the tests to bzlmod in this PR?

The tests use nixpkgs_python_configure which is part of rules_nixpkgs_python and that is currently not available from the BCR. Same situation for rules_haskell_nixpkgs. I'd say let's leave it at that for now.

avdv avatar Jun 14 '24 11:06 avdv

So are we good to merge? I can play arround with a follow up PR for the tests including the other deps as non module dependencies

hofbi avatar Jun 14 '24 11:06 hofbi

So are we good to merge?

CI is still failing. :-(

I can play arround with a follow up PR for the tests including the other deps as non module dependencies

That would be awesome!

avdv avatar Jun 14 '24 12:06 avdv

CI run on ubuntu is now green, but macos is failing with:

Repository rule cc_autoconf defined at:
  /private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/cc_configure.bzl:109:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_tools~cc_configure_extension~local_config_cc':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/cc_configure.bzl", line 91, column 33, in cc_autoconf_impl
		configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 387, column 46, in configure_unix_toolchain
		overriden_tools["ar"] = _find_generic(repository_ctx, "libtool", "LIBTOOL", overriden_tools)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 310, column 32, in _find_generic
		auto_configure_fail(msg)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/lib_cc_configure.bzl", line 112, column 9, in auto_configure_fail
		fail("\n%sAuto-Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: 
Auto-Configuration Error: Cannot find libtool or LIBTOOL; either correct your path or set the LIBTOOL environment variable
ERROR: <builtin>: fetching cc_autoconf rule //:bazel_tools~cc_configure_extension~local_config_cc: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/cc_configure.bzl", line 91, column 33, in cc_autoconf_impl
		configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 387, column 46, in configure_unix_toolchain
		overriden_tools["ar"] = _find_generic(repository_ctx, "libtool", "LIBTOOL", overriden_tools)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 310, column 32, in _find_generic
		auto_configure_fail(msg)
	File "/private/var/tmp/_bazel_runner/597fade3707a30aa94c2bd004f76cc2b/external/bazel_tools/tools/cpp/lib_cc_configure.bzl", line 112, column 9, in auto_configure_fail
		fail("\n%sAuto-Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: 
Auto-Configuration Error: Cannot find libtool or LIBTOOL; either correct your path or set the LIBTOOL environment variable

avdv avatar Jun 14 '24 13:06 avdv

https://github.com/bazelbuild/bazel/commit/3c47b4716e6eca2b8ab06253915ef8d1bdd28038 seems to be relevant to the LIBTOOL issue. I've tried out a Nix provided cc toolchain to avoid this issue here. With that the MacOS build proceeds further, but fails with

ERROR: /Volumes/External/Users/andreas/clodl/tests/BUILD.bazel:109:15: Action clotestbin-cc-closure.zip failed: (Terminated): bash failed: error executing Action command (from target //:clotestbin-cc-closure) /nix/store/wb7cjcvaa31s5saxi7z24qwhq709hbhz-bash/bin/bash -c ... (remaining 4 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
otool: error: 'dyld[60391]: <no uuid> /private/var/tmp/_bazel_andreas/5a72690ff69d3fb0757d7797a2268561/execroot/io_tweag_clodl_tests/bazel-out/darwin_arm64-opt-exec-ST-467dab1ad9da/bin/external/io_tweag_clodl/loader': No such file or directory
Terminated: 15

aherrmann avatar Jun 18 '24 11:06 aherrmann

bazelbuild/bazel@3c47b47 seems to be relevant to the LIBTOOL issue. I've tried out a Nix provided cc toolchain to avoid this issue here. With that the MacOS build proceeds further, but fails with

ERROR: /Volumes/External/Users/andreas/clodl/tests/BUILD.bazel:109:15: Action clotestbin-cc-closure.zip failed: (Terminated): bash failed: error executing Action command (from target //:clotestbin-cc-closure) /nix/store/wb7cjcvaa31s5saxi7z24qwhq709hbhz-bash/bin/bash -c ... (remaining 4 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
otool: error: 'dyld[60391]: <no uuid> /private/var/tmp/_bazel_andreas/5a72690ff69d3fb0757d7797a2268561/execroot/io_tweag_clodl_tests/bazel-out/darwin_arm64-opt-exec-ST-467dab1ad9da/bin/external/io_tweag_clodl/loader': No such file or directory
Terminated: 15

I found https://github.com/envoyproxy/envoy-filter-example/issues/121. From there I get the impression that XCode is required to be installed for some reason.

hofbi avatar Jun 21 '24 09:06 hofbi

otool: error: 'dyld[60391]: /private/var/tmp/_bazel_andreas/5a72690ff69d3fb0757d7797a2268561/execroot/io_tweag_clodl_tests/bazel-out/darwin_arm64-opt-exec-ST-467dab1ad9da/bin/external/io_tweag_clodl/loader': No such file or directory

I'm guessing loader refers to this helper binary.

facundominguez avatar Jun 21 '24 11:06 facundominguez

How should we move on with this? Is there something I could/should try or do you have to clarify the next steps internally?

hofbi avatar Jul 15 '24 12:07 hofbi

Hello @hofbi. At this point I think some investigation is needed to explain why the OSX build is currently failing to find the loader helper. You are welcome to dive into it.

facundominguez avatar Jul 15 '24 18:07 facundominguez

Hello @hofbi. At this point I think some investigation is needed to explain why the OSX build is currently failing to find the loader helper. You are welcome to dive into it.

I have no OSX hardware available so can only do debugging in CI, but will try my best

hofbi avatar Jul 15 '24 18:07 hofbi