ispc Tests fail if llvm-lit `not` command is missing

make check-all reports a significant amount of failing test cases with LLVM 14.0.6:

Failing Tests (25):
    ispc :: 1447.ispc
    ispc :: 1505.ispc
    ispc :: 1588_stdin.ispc
    ispc :: 1596.ispc
    ispc :: 1654.ispc
    ispc :: 1688.ispc
    ispc :: 1739.ispc
    ispc :: 1849.ispc
    ispc :: 1872.ispc
    ispc :: 2198.ispc
    ispc :: 2280.ispc
    ispc :: 861.ispc
    ispc :: alloca_error.ispc
    ispc :: arg_parsing_errors.ispc
    ispc :: error_undef_symbol.ispc
    ispc :: float_literals_errors.ispc
    ispc :: intrinsic_target_specific.ispc
    ispc :: llvm_ident.ispc
    ispc :: noinline_error.ispc
    ispc :: pragma_unroll_errors.ispc
    ispc :: pragma_unroll_errors_2.ispc
    ispc :: print_errors.ispc
    ispc :: response_files.ispc
    ispc :: scoped-typedef.ispc
    ispc :: varying_init_error.ispc

  Expected Passes    : 104
  Unsupported Tests  : 29
  Unexpected Failures: 25

not is an auxiliary helper command for lit, which is not typically available as a regular command in path

https://github.com/llvm/llvm-project/blob/main/llvm/utils/not/not.cpp

Oct 20 '22 01:10 StefanBruens

Hi @StefanBruens , We expect that lit-tests are run using the same LLVM as for ISPC build. LLVM for ISPC should be built using alloy.py script which will patch it and include all necessary tools for ISPC development/testing to LLVM installation. Please refer to this doc: https://github.com/ispc/ispc/wiki/ISPC-Development-Guide#1_Building_LLVM on how to build LLVM.

Oct 26 '22 17:10 aneshlya

There is no actual requirement for a patched, non-upstream version of LLVM when using a current LLVM. Running the tests during build should be sufficient to verify a suitable LLVM version.

Both Fedora and openSUSE use the generic distribution LLVM for ISPC. (*BSD also seems to expect the system LLVM to work, see e.g. https://github.com/ispc/ispc/issues/2318).

Having a separate "special" LLVM for each and every package which requires LLVM does not scale, it costs compile time, disk space (mirrors and target system), delays security fixes, and so much more.

Oct 31 '22 10:10 StefanBruens

Using generic LLVM for ISPC build "mostly works", meaning:

we don't have control over patches, meaning we don't know if all the issues we care about are fixed. For 14.0 and later we don't have patches currently, but for older LLVM we do. We tend to find bugs as we go and add patches over time.
we don't have control over options that are used to build LLVM and that may trigger unexpected issues that we are not aware of. Basically it means building "production" version that is not tested well.
we don't have control over enabled targets in LLVM, so this limits the targets available in ISPC.
lit tests are mostly "sanity testing", not a thorough testing.
to be able to test ISPC properly you need to follow recommendations for "development environment" - i.e. LLVM being built in proper configuration, etc.

LLVM is fundamentally is a core part of ISPC backend and it's essential that it's configured and built as we expect it. Otherwise we have no way to guarantee that resulting ISPC build works as expected.

Oct 31 '22 23:10 dbabokin

I can´t see anything which supports the exclusive usage of a private libLLVM just for ISPC.

ISPC build with a current libLLVM 14 or 15 has no known bugs. In case you discover new bugs, these are hopefully also fixed upstream. In case the unfixed libLLVM is kept, ISPC does not improve, but it is no worse than the old state. On the other hand, if bugs are fixed upstream, using a system wide libLLVM also fixes the bugs in ISPC without further action.
You will never have control over all options. Even with a private libLLVM it may be compiled with different host compilers, with different system libraries compiled with different options, running on different kernels on different CPUs. All these may introduce issues, and the best way to catch these is to add more tests.
Most people are not interested in any targets but "native", any maybe webassembly. Neither aarch32 nor x86 will be used much on 64bit archs (current aarch64 CPUs can't even execute aarch32 any longer), so these targets are for most parts untested in production.
lit tests increase test coverage, and we would like to enable these in addition to the regular test suite. Is there anything else which would testing "thorough"? I don´t see any mention of LLVMs own test suite in the build instructions - wouldn't it be better if these were also run? A system libLLVM including the test suite actually has a better test coverage than ISPCs build, as far as I can see.

Using a system wide libLLVM results in a significantly better coverage of its code, i.e. code paths with are unused by ISPCs test suite, but are used by other libLLVM consumers and also by ISPC depending on the input source code. This does not guarantee correctness, but as computer science tells us, correctness is also not "guaranteed" by a private libLLVM.

Nov 01 '22 22:11 StefanBruens

@StefanBruens, I need to notice that not is the build-time dependency of ISPC. If you want to run make check-all you should install it in your environment. The project CI provides it as a part of self-build LLVM toolchain. Arch Linux provides it inside llvm package https://archlinux.org/packages/extra/x86_64/llvm/. Alpine Linux provides it inside llvm-test-utils package https://pkgs.alpinelinux.org/contents?branch=edge&name=llvm14-test-utils&arch=x86_64&repo=main.

As I understand, it is openSUSE in your case, and, as I see here https://build.opensuse.org/package/view_file/openSUSE:Factory/llvm14/llvm14.spec?expand=1 (lines 1653-1667), llvm-devel does not ship not utility as it does, e.g., with FileCheck which is also an auxiliary helper command needed for lit.

If there is no chance to provide not utility then you may just skip make check-all stage as, e.g., Fedora does https://src.fedoraproject.org/rpms/ispc/blob/rawhide/f/ispc.spec#_101.

Mar 29 '23 20:03 nurmukhametov