[SYCL][CUDA] Enable CXX standard library funcs for CUDA backend

[hdelan]

This PR allows CXX stdlib funcs to be used for NVPTX backend.

See https://github.com/intel/llvm/discussions/6379

llvm-test-suite test: https://github.com/intel/llvm-test-suite/pull/1112

It also adds the compiler flag "-fbundle-no-offload-arch", which allows device code bundles to omit arch information, meaning a bundle with the tag sycl-nvptx64-nvidia-cuda-sm_50 can be bundled with the tag sycl-nvptx64-nvidia-cuda when the flag is enabled. This allows for generic bc bundles for a given backend.

I have not moved the NVPTX complex implementation to libdevice. Let me know if I should do so.

The behaviour of IsMathErrnoDefault() now changes so that -fmath-errno is enabled by default for the CUDA toolchain. This allows libdevice funcs to be used by a similar exploit that the SYCL toolchain uses. If necessary this could be reset once the upstream discussion on NVPTX codegen for llvm.sin and other llvm builtins has been resolved and implemented. See https://discourse.llvm.org/t/nvptx-codegen-for-llvm-sin-and-friends/58170/11?u=jdoerfert

cc @Naghasan @AerialMantis @steffenlarsen @bader @jdoerfert

[hdelan]

/verify with https://github.com/intel/llvm-test-suite/pull/1112

[hdelan]

Can I get the logs for the llvm test suite failures please?

[steffenlarsen]

Failure during build:

[2870/2938] Generating ../../lib/libsycl-fallback-complex.o
FAILED: lib/libsycl-fallback-complex.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda .../LLVM_Test_Suite/llvm.src/libdevice/fallback-complex.cpp -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-complex.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2871/2938] Generating ../../lib/libsycl-itt-user-wrappers.o
FAILED: lib/libsycl-itt-user-wrappers.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda .../LLVM_Test_Suite/llvm.src/libdevice/itt_user_wrappers.cpp -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-itt-user-wrappers.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2872/2938] Generating ../../lib/libsycl-complex.o
FAILED: lib/libsycl-complex.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda .../LLVM_Test_Suite/llvm.src/libdevice/complex_wrapper.cpp -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-complex.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2873/2938] Generating ../../lib/libsycl-fallback-cassert.o
FAILED: lib/libsycl-fallback-cassert.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda -fno-sycl-instrument-device-code .../LLVM_Test_Suite/llvm.src/libdevice/fallback-cassert.cpp -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-cassert.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2874/2938] Generating ../../lib/libsycl-fallback-cmath.o
FAILED: lib/libsycl-fallback-cmath.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda .../LLVM_Test_Suite/llvm.src/libdevice/fallback-cmath.cpp -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-cmath.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2875/2938] Generating ../../lib/libsycl-fallback-imf.o
FAILED: lib/libsycl-fallback-imf.o 
cd .../LLVM_Test_Suite/llvm.obj/tools/libdevice && .../LLVM_Test_Suite/llvm.obj/bin/clang-15 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda .../LLVM_Test_Suite/llvm.obj/lib/libdevice/imf_fp32_fallback.cpp -I .../LLVM_Test_Suite/llvm.src/libdevice/imf -o .../LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-imf.o
clang-15: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
[2876/2938] Building CXX object tools/sycl/tools/sycl-trace/CMakeFiles/sycl_pi_trace_collector.dir/pi_trace_collector.cpp.o
[2877/2938] Generating ../../lib/libsycl-fallback-cmath.spv
clang-15: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
clang-15: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
[2878/2938] Generating ../../lib/fallback-imf-fp64-host.o
[2879/2938] Generating ../../lib/fallback-imf-fp32-host.o
[2880/2938] Generating ../../lib/libsycl-fallback-complex.spv
clang-15: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
clang-15: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
[2881/2938] Generating ../../lib/libsycl-fallback-imf.spv
clang-15: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
clang-15: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File ".../LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 71, in <module>
    ret = main()
  File ".../LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 67, in main
    return do_compile(args)
  File ".../LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 42, in do_compile
    subprocess.check_call(cmake_cmd, cwd=abs_obj_dir)
  File "/rdrive/ics/itools/pkgtools/python/v3_7_4/efi2_rhel72/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.../LLVM_Test_Suite/llvm.obj', '--', 'deploy-sycl-toolchain', '-j', '12']' returned non-zero exit status 1.
get-build-results ERROR: error running compile script
get-build-results INFO: === stage get-build-results end ===
  File ".../LLVM_Test_Suite/llvm_ci/intel/worker/tools/common/base.py", line 472, in _get_build_results
    options.pr_number, os.getcwd(), options.build_dir, self.logger)
  File ".../LLVM_Test_Suite/llvm_ci/intel/worker/tools/common/compile.py", line 51, in do_compile
    raise Exception("error running compile script")

[hdelan]

Failure during build:

Thanks @steffenlarsen

[hdelan]

/verify with https://github.com/intel/llvm-test-suite/pull/1112

[hdelan]

Sorry to bother again but would be great if someone could post the logs for llvm-test-suite. Thanks in advance

[bader]

Sorry to bother again but would be great if someone could post the logs for llvm-test-suite. Thanks in advance

 FAILED: lib/libsycl-fallback-complex.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/fallback-complex.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-complex.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2877/2944] Generating ../../lib/libsycl-itt-stubs.o
 FAILED: lib/libsycl-itt-stubs.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/itt_stubs.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-itt-stubs.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2878/2944] Generating ../../lib/libsycl-cmath.o
 FAILED: lib/libsycl-cmath.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/cmath_wrapper.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-cmath.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2879/2944] Generating ../../lib/libsycl-fallback-cmath-fp64.o
 FAILED: lib/libsycl-fallback-cmath-fp64.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/fallback-cmath-fp64.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-cmath-fp64.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2880/2944] Generating ../../lib/libsycl-fallback-cstring.o
 FAILED: lib/libsycl-fallback-cstring.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/fallback-cstring.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-cstring.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2881/2944] Generating ../../lib/libsycl-itt-user-wrappers.o
 FAILED: lib/libsycl-itt-user-wrappers.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/itt_user_wrappers.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-itt-user-wrappers.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2882/2944] Generating ../../lib/libsycl-fallback-imf-fp64.o
 FAILED: lib/libsycl-fallback-imf-fp64.o 
 cd /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/tools/libdevice && /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/bin/clang-16 -fsycl -c -I /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/libdevice/imf -Wno-sycl-strict -Wno-undefined-internal -sycl-std=2020 -fno-sycl-libspirv -fno-bundle-offload-arch -fsycl-targets=spir64_x86_64-unknown-unknown,spir64_gen-unknown-unknown,spir64_fpga-unknown-unknown,spir64-unknown-unknown,nvptx64-nvidia-cuda /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libdevice/imf_fp64_fallback.cpp -o /netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj/lib/libsycl-fallback-imf-fp64.o
 clang-16: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice
 [2883/2944] Generating ../../lib/libsycl-fallback-cmath-fp64.spv
 clang-16: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
 clang-16: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
 [2884/2944] Building CXX object tools/sycl/tools/sycl-trace/CMakeFiles/sycl_pi_trace_collector.dir/pi_trace_collector.cpp.o
 [2885/2944] Generating ../../lib/libsycl-fallback-complex.spv
 clang-16: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
 clang-16: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
 [2886/2944] Generating ../../lib/libsycl-fallback-complex-fp64.spv
 clang-16: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
 clang-16: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
 [2887/2944] Generating ../../lib/libsycl-fallback-imf.spv
 clang-16: warning: argument unused during compilation: '-fno-sycl-libspirv' [-Wunused-command-line-argument]
 clang-16: warning: argument unused during compilation: '-fno-bundle-offload-arch' [-Wunused-command-line-argument]
 ninja: build stopped: subcommand failed.
 Traceback (most recent call last):
   File "/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 71, in <module>
     ret = main()
   File "/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 67, in main
     return do_compile(args)
   File "/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.src/buildbot/compile.py", line 42, in do_compile
     subprocess.check_call(cmake_cmd, cwd=abs_obj_dir)
   File "/rdrive/ics/itools/pkgtools/python/v3_7_4/efi2_rhel72/lib/python3.7/subprocess.py", line 347, in check_call
     raise CalledProcessError(retcode, cmd)
 subprocess.CalledProcessError: Command '['cmake', '--build', '/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm.obj', '--', 'deploy-sycl-toolchain', '-j', '12']' returned non-zero exit status 1.
 [2022-08-01 17:41:18,865] get-build-results ERROR: error running compile script
 [2022-08-01 17:41:18,866] get-build-results INFO: === stage get-build-results end ===
   File "/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm_ci/intel/worker/tools/common/base.py", line 472, in _get_build_results
     options.pr_number, os.getcwd(), options.build_dir, self.logger)
   File "/netbatch/donb1911550_00/runDir/jenkins-dir/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite/llvm_ci/intel/worker/tools/common/compile.py", line 51, in do_compile
     raise Exception("error running compile script")

[bader]

On Windows

 FAILED: tools/sycl/source/CMakeFiles/sycl_object.dir/detail/pi.cpp.obj 
 C:\PROGRA~2\MICROS~2\2019\PROFES~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\cl.exe  /nologo /TP -DCL_TARGET_OPENCL_VERSION=220 -DSYCL2020_DISABLE_DEPRECATION_WARNINGS -DUNICODE -DXPTI_ENABLE_INSTRUMENTATION -DXPTI_STATIC_LIBRARY -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__SYCL_BUILD_SYCL_DLL -D__SYCL_INTERNAL_API -Itools\sycl\source -IW:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source -Iinclude -IW:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\llvm\include -IW:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\xpti\include -IW:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\include -I_deps\ocl-headers-src /WX /W4 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4324 -w14062 -we4238 /Gw /EHsc /Zi  /O2 /Ob2   -wd4996 -UNDEBUG /MD -std:c++17 /showIncludes /Fotools\sycl\source\CMakeFiles\sycl_object.dir\detail\pi.cpp.obj /Fdtools\sycl\source\CMakeFiles\sycl_object.dir\ /FS -c W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp
 C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\iostream(39): error C2370: 'std::cin': redefinition; different storage class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.obj\include\sycl/detail/iostream_proxy.hpp(6): note: see declaration of 'std::cin'
 C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\iostream(40): error C2370: 'std::cout': redefinition; different storage class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.obj\include\sycl/detail/iostream_proxy.hpp(7): note: see declaration of 'std::cout'
 C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\iostream(41): error C2370: 'std::cerr': redefinition; different storage class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.obj\include\sycl/detail/iostream_proxy.hpp(8): note: see declaration of 'std::cerr'
 C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include\iostream(42): error C2370: 'std::clog': redefinition; different storage class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.obj\include\sycl/detail/iostream_proxy.hpp(9): note: see declaration of 'std::clog'
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(386): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(398): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(408): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(450): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(539): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(590): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(593): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(594): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(595): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(596): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(597): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(599): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(601): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(603): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(606): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(607): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(608): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(613): error C2088: '<<': illegal for class
 W:\jenkins-dir\workspace\SYCL_CI\intel\Win\Test_Suite\llvm.src\sycl\source\detail\pi.cpp(618): error C2088: '<<': illegal for class

EDIT: this is not related to the change. I recall that Windows build is already broken.

[aelovikov-intel]

I'm working on Windows build.

[hdelan]

Thanks @bader. Are the CI scripts for Jenkins/llvm-test-suite available somewhere online? Would make it easier for me to debug.

[bader]

Are the CI scripts for Jenkins/llvm-test-suite available somewhere online?

AFAIK, no. It should be the same as GitHub actions scripts https://github.com/intel/llvm/blob/sycl/.github/workflows/sycl_linux_build_and_test.yml#L194, but instead of pulling the head of the intel branch, it pulls the head of the PR specified in the comment. Please, let me know if you are unable to reproduce the problem.

@DoyleLi is working on making the logs available.

[hdelan]

AFAIK, no. It should be the same as GitHub actions scripts https://github.com/intel/llvm/blob/sycl/.github/workflows/sycl_linux_build_and_test.yml#L194, but instead of pulling the head of the intel branch, it pulls the head of the PR specified in the comment. Please, let me know if you are unable to reproduce the problem.

@DoyleLi is working on making the logs available.

Thanks @bader . Also can I ask: is there a CUDA installation on the CI machine that does the Jenkins build? Building the libdevice objects requires normal nvptx backend -c compilation. However the log is saying that it cannot find libdevice.bc, meaning either the CUDA_TOOLKIT_ROOT_DIR is not set or that there is simply no CUDA installation. Let me know your thoughts. Thanks

[bader]

is there a CUDA installation on the CI machine that does the Jenkins build? Building the libdevice objects requires normal nvptx backend -c compilation. However the log is saying that it cannot find libdevice.bc, meaning either the CUDA_TOOLKIT_ROOT_DIR is not set or that there is simply no CUDA installation.

It's not clear from the logs and another suspicious thing I noticed is that one of the internal scripts is configured w/o cuda support. @DoyleLi, could you answer the question from @hdelan and check that internal scripts are configured correctly, please?

[DoyleLi]

AFAIK, no. It should be the same as GitHub actions scripts https://github.com/intel/llvm/blob/sycl/.github/workflows/sycl_linux_build_and_test.yml#L194, but instead of pulling the head of the intel branch, it pulls the head of the PR specified in the comment. Please, let me know if you are unable to reproduce the problem. @DoyleLi is working on making the logs available.

Thanks @bader . Also can I ask: is there a CUDA installation on the CI machine that does the Jenkins build? Building the libdevice objects requires normal nvptx backend -c compilation. However the log is saying that it cannot find libdevice.bc, meaning either the CUDA_TOOLKIT_ROOT_DIR is not set or that there is simply no CUDA installation. Let me know your thoughts. Thanks

Hi @hdelan. No, the llvm_test_suite check on Linux is running on system without CUDA installed on it. So I think maybe the patch should consider the case that compiling intel/llvm source code on system without CUDA. FYI, here is cmake and build commands when we compile intel/llvm:

 cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=X86;NVPTX -DLLVM_EXTERNAL_PROJECTS=sycl;llvm-spirv;opencl;libdevice;xpti;xptifw -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=${WORKSPACE}/llvm.src/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=${WORKSPACE}/llvm.src/llvm-spirv -DLLVM_EXTERNAL_XPTI_SOURCE_DIR=${WORKSPACE}/llvm.src/xpti -DXPTI_SOURCE_DIR=${WORKSPACE}/llvm.src/xpti -DLLVM_EXTERNAL_XPTIFW_SOURCE_DIR=${WORKSPACE}/llvm.src/xptifw -DLLVM_EXTERNAL_LIBDEVICE_SOURCE_DIR=${WORKSPACE}/llvm.src/libdevice -DLLVM_ENABLE_PROJECTS=clang;sycl;llvm-spirv;opencl;libdevice;xpti;xptifw;clang-tools-extra;compiler-rt;libclc -DLIBCLC_TARGETS_TO_BUILD=;amdgcn--;amdgcn--amdhsanvptx64--;nvptx64--nvidiacl -DLIBCLC_GENERATE_REMANGLED_VARIANTS=OFF -DSYCL_BUILD_PI_HIP_PLATFORM=AMD -DLLVM_BUILD_TOOLS=ON -DSYCL_ENABLE_WERROR=ON -DCMAKE_INSTALL_PREFIX=${WORKSPACE}/llvm.obj/install -DSYCL_INCLUDE_TESTS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DLLVM_ENABLE_SPHINX=OFF -DBUILD_SHARED_LIBS=OFF -DSYCL_ENABLE_XPTI_TRACING=ON -DLLVM_ENABLE_LLD=OFF -DXPTI_ENABLE_WERROR=ON -DSYCL_CLANG_EXTRA_FLAGS= -DSYCL_ENABLE_PLUGINS=level_zero;opencl ${WORKSPACE}/llvm.src/llvm
 

 cmake --build ${WORKSPACE}/llvm.obj -- deploy-sycl-toolchain -j 12

[bader]

@hdelan, do I understand it correctly, that this change adds dependency on CUDA SDK to build libclc for NVPTX? I guess today we use clang to emit NVVM, which doesn't require SDK for building the toolchain. SDK is required only when we use the compiler to compile for NVIDIA GPU target.

[hdelan]

@hdelan, do I understand it correctly, that this change adds dependency on CUDA SDK to build libclc for NVPTX? I guess today we use clang to emit NVVM, which doesn't require SDK for building the toolchain. SDK is required only when we use the compiler to compile for NVIDIA GPU target.

Thanks @DoyleLi @bader . I have removed the dependency on CUDA SDK so this should work now.

[bader]

Okay. Thanks. Let's test.

[bader]

/verify with https://github.com/intel/llvm-test-suite/pull/1112

[hdelan]

Failing still. Could someone post the logs please? Thanks

[bader]

Tests pass on Linux, but failed on Windows. I think failures are not related to your patch and probably will be fixed by https://github.com/intel/llvm/pull/6517. @aelovikov-intel, am I right?

[aelovikov-intel]

Tests pass on Linux, but failed on Windows. I think failures are not related to your patch and probably will be fixed by #6517. @aelovikov-intel, am I right?

Hopefully so.

[hdelan]

Ok good news then. Let me know if there are any changes you think I should make to this patch. The other failing tests I do not believe are related to this PR.

[bader]

The other failing tests I do not believe are related to this PR.

Agree. @npmiller took care about the failures on HIP - see https://github.com/intel/llvm/pull/6519#issuecomment-1203755950. Could you do the same for CUDA, please?

[hdelan]

Could you do the same for CUDA, please?

Yep here you go https://github.com/intel/llvm-test-suite/pull/1127

[bader]

@intel/llvm-reviewers-cuda, @intel/llvm-reviewers-runtime, ping.

[hdelan]

ping @jinge90

[jinge90]

ping @jinge90

Hi, @hdelan The libdevice change looks OK and I have 2 questions:

1. Could you also include related libdevice functions in this PR, they are used to support std::complex in SYCL device programming.
2. For assert support, it seems that we finally rely on CUDA runtime to handle "__assertfail", am I correct?

Thanks very much.

[hdelan]

1. Could you also include  related libdevice functions in this PR, they are used to support std::complex in SYCL device programming.

2. For assert support, it seems that we finally rely on CUDA runtime to handle "__assertfail", am I correct?

Thanks for response.

1. In discussions we have sort of decided that the current approach to std::complex is OK and we don't need to implement std::complex through libdevice. @bader please confirm.
2. Yes this is correct. This specialization is happening in libdevice since in earlier discussions we concluded that assertfail doesn't belong in libclc, where spirv or other builtins are specialized for PTX backend if compiling for that triple.

Let me know if you have any other questions

[jinge90]

OK, thanks for the clarification.

[steffenlarsen]

@jinge90 & @smanna12 - Friendly ping.