wanted: llvm/llvm-project

[aaronmondal]

Users working with (or on) LLVM may require very recent versions of the repo. We have module patches for the llvm-bazel-overlay but found that it would be better to add a module here first before upstreaming patches to LLVM main.

Would it be OK for us to add a module with patches here so that we can upstream the patches later to the main LLVM repo? We would want to update such a module probably around ~bi-weekly, tagged like 16.0.0-20221005.0-asdfasdf.

A big question is how to not hammer CI with such a module. We would only implement the testing infrastructure that is part of the llvm-project-overlay, not the entire LLVM test infrastructure. We should also keep in mind that the llvm-project-overlay is ~5 projects in one. However, running the small subset of the LLVM tests would still likely be the largest CI job out of all the modules that are currently part of the bazel-central-registry. My initial naive estimation for running that small test subset would be ~500 single-core build minutes per architecture/OS combination :sweat_smile: Maybe only building a single target (probably the @llvm-project//clang:clang target) and only building for x86_64 Linux initially would be an option?

[fmeum]

--nobuild may be appealing here. Most bzlmod-specific issues and changes to the Bazel interface of the module will probably be caught in the analysis phase. That plus a small real build on a single platform could be a decent starting point.

[aaronmondal]

Patches still WIP. Started upstreaming to LLVM as well. For those interested, a "buildable" variant is now at https://github.com/eomii/bazel-eomii-registry/tree/llvm-project-overlay/modules/llvm-project-overlay, which can be enabled by adding

build --registry=https://raw.githubusercontent.com/eomii/bazel-eomii-registry/llvm-project-overlay/

to .bazelrc. The toolchain is buildable, but a few clang headers, are missing (easy fix, just didn't get around to it yet) and I think i messed up some if-else clauses around the custom commit logic (easy fix as well). Things should more or less work though.

My proposition is to name this module @llvm-project-overlay with version 16.0.0-bcr.0. The variant I'm upstreaming to LLVM will support in-tree and out-of-tree builds using --experimental_enable_bzlmod, as well as custom patching in downstream repos. I'm adding some logic for hopefully good forward-compatibility, so it will probably not be necessary to update this module too often.

WIP commit message explaining a little bit more what is going on:

This commit adds the llvm-project to the BCR.

Since we require intermediary setup repositories, the importable module
is called llvm-project-overlay. The workspace label that users will
interact with is `@llvm-project`, which is made available via use_repo.

The changes in bzlmod_compatibility_patch.diff modify the existing
overlay in the LLVM project while being backwards-compatible with the
WORKSPACE workflow. The changes in the bcr_compatibility_patch.diff
add the actual MODULE.bazel and extensions. These have been built in a
way that builds should run both from within the llvm/utils/bazel
directory once these changes are upstreamed to LLVM, as well as being
configurable from downstream modules when this module is used as a
dependency coming from the BCR.

First, a MODULE.bazel, an empty WORKSPACE and an empty BUILD.bazel file
are patched in on the BCR side. This preliminary repository is available
as @llvm-raw. Then we invoke extensions in @llvm-raw//:MODULE.bazel to
apply the Bazel overlay already present in the LLVM sources. The
resulting repository is available as `@llvm-project`. Common sources
between @llvm-raw and `@llvm-project` are either symlinked or treated as
independent repositories, depending on the extension configuration.

Users working with LLVM will often require the flexibility of applying
patches to the LLVM sources, so we provide two options of configuring the
sources:

1. Using the vanilla sources at the commit specified in the BCR release.
   This variant will not allow custom patches, but will reuse the already
   fetched LLVM sources as a symlinked local repository. This is efficient
   and will not require any additional downloads. With this method most of
   the `@llvm-project` sources will be symlinks to files in @llvm-raw.

2. Using a custom commit. This variant will download a copy of the LLVM
   sources at the specified commit. Users may apply custom patches to
   these sources. While this requires a second download of the sources,
   it makes the BCR module very future-proof. With this method the
   Bazel-specific code in @llvm-raw//utils/bazel is reused to download and
   configure `@llvm-project` as an independent repository. As long as bzlmod
   support has not been upstreamed, this method will require re-applying
   (potentially conflict-resolved variants of) the BCR patches to maintain
   bzlmod compatibility.

[mattyclarkson]

We are also interested in getting LLVM in the BCR. Having a look at getting something upstreamed as a PR.