Added support clang 14+ new directory structure

[mboisson]

trafficstars

[mboisson]

Don't merge yet, this is not the right fix

[mboisson]

:( apparently, this never got merged https://github.com/easybuilders/easybuild-easyblocks/pull/2264 and part of it is included in this PR.... can we go back and merge the above ?

[mboisson]

In fact, I am not able to get clang to build without downloading the full package llvm-project-%(version)s.src.tar.xz. The sum of components does not equal the whole thing, there are parts missing, see https://github.com/llvm/llvm-project/issues/58320 .

I would suggest to drop the per-component download, and to select which component to install based on LLVM_ENABLE_PROJECTS

[ocaisa]

@mboisson This looks good, but there seems to be a conflict right now, can you fix that?

Is there a PR that uses this? I can add one based on your repos.

I'd like to do one which includes flang from v15 up. Can we add that to the sanity check step?

        if 'flang' in self.cfg['llvm_projects'] and LooseVersion(self.version) >= LooseVersion('15'):
            custom_paths['files'].extend(["bin/flang"])

[mboisson]

@mboisson This looks good, but there seems to be a conflict right now, can you fix that?

I have no clue how to merge that conflict... the code is completely different and is not in a single commit so I can't cherry-pick the change :( You will have to fix the conflict while merging

[mboisson]

@ocaisa the other changes are incorporated

[mboisson]

I am not sure why the tests are not actually running... Any idea @ocaisa ?

[mboisson]

I tried to resolve the conflicts by cherry-picking all missing commits to this file and manually fixing the conflict when cherry-picking the commit that caused it, but Github still does not like it. A maintainer will have to fix it when merging. @ocaisa

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-13.0.1-GCCcore-11.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) login1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/85fc28ed1e4a6e4eb273f5e3e9e93ec9 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• FAIL (build issue) Clang-11.0.1-GCCcore-10.2.0.eb (partial log available at https://gist.github.com/63ee9dc613f8649e22750d30d43ea680)
• SUCCESS Clang-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 2 (2 easyconfigs in total) cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, Python 3.6.8 See https://gist.github.com/9e932dfd1536a802a52dd5cb4e6f1ca4 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• FAIL (build issue) Clang-15.0.4-GCCcore-11.3.0.eb (partial log available at https://gist.github.com/fe7624019f4b83e7a4c1455defc86e54)

Build succeeded for 0 out of 1 (1 easyconfigs in total) cnx2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/ca9aa52c18f12124cd60c2d11d414dd0 for a full test report.

[ocaisa]

The failing test for Clang 11 is related to the sanity check, there is no lib/libomptarget.rtl.x86_64.so. That's not related to this PR though, so no need to fix here.

The failure for Clang 15 is partially related to the fact the compiler is called flang-new but also to

no file found at 'lib/libc++.so' in /home/ocaisa/.local/easybuild/software/Clang/15.0.4-GCCcore-11.3.0
no file found at 'lib/libc++abi.so' in /home/ocaisa/.local/easybuild/software/Clang/15.0.4-GCCcore-11.3.0

Looking at the configure step I see for Clang 15 I see:

CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libcxx is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libcxx or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.


CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libcxxabi is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libcxxabi or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.


CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libunwind is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libunwind or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.

[mboisson]

The failing test for Clang 11 is related to the sanity check, there is no lib/libomptarget.rtl.x86_64.so. That's not related to this PR though, so no need to fix here.

The failure for Clang 15 is partially related to the fact the compiler is called flang-new but also to

no file found at 'lib/libc++.so' in /home/ocaisa/.local/easybuild/software/Clang/15.0.4-GCCcore-11.3.0
no file found at 'lib/libc++abi.so' in /home/ocaisa/.local/easybuild/software/Clang/15.0.4-GCCcore-11.3.0

Looking at the configure step I see for Clang 15 I see:

CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libcxx is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libcxx or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.


CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libcxxabi is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libcxxabi or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.


CMake Warning at CMakeLists.txt:101 (message):
  Using LLVM_ENABLE_PROJECTS=libunwind is deprecated now, please use
  -DLLVM_ENABLE_RUNTIMES=libunwind or see the instructions at
  https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.

when did that become deprecated ?! I built 15.0.2 without issue

[mboisson]

🤔 I do have these warnings as well, but somehow the sanity checks did not fail for my build, even though those libraries are indeed missing. I'll need to investigate (probably next week)

[mboisson]

== 2022-10-14 06:28:44,906 build_log.py:266 INFO sanity checking...
== 2022-10-14 06:28:44,908 easyblock.py:3820 INFO Starting sanitycheck step
== 2022-10-14 06:28:44,909 easyblock.py:3828 INFO Running method sanity_check_step part of step sanitycheck
== 2022-10-14 06:28:44,909 easyblock.py:3007 INFO Running sanity check with extra modules: python/3.8
== 2022-10-14 06:28:44,910 easyblock.py:3243 INFO Using (only) sanity check paths specified by easyconfig file: {'dirs': ['lib/python3.8/site-packages'], 'files': []}
== 2022-10-14 06:28:44,910 easyblock.py:3291 INFO Using customised sanity check commands: ['clang --help', 'clang++ --help', 'llvm-config --cxxflags', "python -c 'import clang'"]
== 2022-10-14 06:29:08,831 easyblock.py:3484 INFO Loading extra modules for sanity check: python/3.8
== 2022-10-14 06:29:09,251 easyblock.py:3506 INFO sanity check command clang --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:09,285 easyblock.py:3506 INFO sanity check command clang++ --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:09,296 easyblock.py:3506 INFO sanity check command llvm-config --cxxflags ran successfully! (output: -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/clang/15.0.2/include -std=c++14    -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
== 2022-10-14 06:29:10,026 easyblock.py:3506 INFO sanity check command python -c 'import clang' ran successfully! (output: )
== 2022-10-14 06:29:11,021 easyblock.py:3007 INFO Running sanity check with extra modules: python/3.9
== 2022-10-14 06:29:11,021 easyblock.py:3243 INFO Using (only) sanity check paths specified by easyconfig file: {'dirs': ['lib/python3.9/site-packages'], 'files': []}
== 2022-10-14 06:29:11,021 easyblock.py:3291 INFO Using customised sanity check commands: ['clang --help', 'clang++ --help', 'llvm-config --cxxflags', "python -c 'import clang'"]
== 2022-10-14 06:29:19,531 easyblock.py:3484 INFO Loading extra modules for sanity check: python/3.9
== 2022-10-14 06:29:19,569 easyblock.py:3506 INFO sanity check command clang --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:19,602 easyblock.py:3506 INFO sanity check command clang++ --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:19,613 easyblock.py:3506 INFO sanity check command llvm-config --cxxflags ran successfully! (output: -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/clang/15.0.2/include -std=c++14    -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
== 2022-10-14 06:29:20,835 easyblock.py:3506 INFO sanity check command python -c 'import clang' ran successfully! (output: )
== 2022-10-14 06:29:21,766 easyblock.py:3007 INFO Running sanity check with extra modules: python/3.10
== 2022-10-14 06:29:21,766 easyblock.py:3243 INFO Using (only) sanity check paths specified by easyconfig file: {'dirs': ['lib/python3.10/site-packages'], 'files': []}
== 2022-10-14 06:29:21,766 easyblock.py:3291 INFO Using customised sanity check commands: ['clang --help', 'clang++ --help', 'llvm-config --cxxflags', "python -c 'import clang'"]
== 2022-10-14 06:29:30,818 easyblock.py:3484 INFO Loading extra modules for sanity check: python/3.10
== 2022-10-14 06:29:30,852 easyblock.py:3506 INFO sanity check command clang --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:30,884 easyblock.py:3506 INFO sanity check command clang++ --help ran successfully! (output: OVERVIEW: clang LLVM compiler
== 2022-10-14 06:29:30,894 easyblock.py:3506 INFO sanity check command llvm-config --cxxflags ran successfully! (output: -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/clang/15.0.2/include -std=c++14    -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
== 2022-10-14 06:29:30,923 easyblock.py:3506 INFO sanity check command python -c 'import clang' ran successfully! (output: )

Oh! Our python hook injects sanity_check_paths... and apparently the EasyBlock does not do internal checks if that is defined.... 🤔

Why is it not adding to sanity_check_paths rather than ignoring its own ?

[mboisson]

Ah, I guess we should set enhance_sanity_check in our hooks

[mboisson]

I'm not sure if using LLVM_ENABLE_RUNTIMES works or not. See https://github.com/llvm/llvm-project/issues/59068

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• FAIL (build issue) Clang-15.0.5-GCCcore-11.3.0.eb (partial log available at https://gist.github.com/7a1095e350d35c7b99c66de94df4e8a0)

Build succeeded for 0 out of 1 (1 easyconfigs in total) cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, Python 3.6.8 See https://gist.github.com/808ddb50353e531bd7ec143ef5a8d970 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-15.0.5-GCCcore-11.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) login1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/12b6dfe035a13bc3afb29eb6fb2a93ae for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-13.0.1-GCCcore-11.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) login1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/af457d88f14a98778665d5198d4c60dd for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-13.0.1-GCCcore-11.2.0-CUDA-11.4.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, Python 3.6.8 See https://gist.github.com/30074b89aa9e6d4a981425783a822b46 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) cnx2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/4ec3268d6cd33c8c1ade3b432a950111 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-11.0.1-GCCcore-10.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) cnx4 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/e8e353295db6af73e59a3d10c769c807 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, Python 3.6.8 See https://gist.github.com/f975e73b47250690c6927ef0fdd3fed6 for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-13.0.1-GCCcore-11.2.0-CUDA-11.4.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total) cnx2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/c721ea8c70087972f3f580efd3f048ea for a full test report.

[ocaisa]

Test report by @ocaisa

Overview of tested easyconfigs (in order)

• SUCCESS Clang-13.0.1-GCCcore-11.2.0-CUDA-11.4.1.eb
• SUCCESS Clang-11.0.1-GCCcore-10.2.0.eb
• SUCCESS Clang-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total) login1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, Python 3.6.8 See https://gist.github.com/e860f312fedad6fe181042367bc359dc for a full test report.

[ocaisa]

Last test was to ensure --module-only was also now working for Clang

[mboisson]

Thanks for all the hard work @mboisson !

🥳