clvk icon indicating copy to clipboard operation
clvk copied to clipboard

RPI 4B Broadcom Videocore 6 Vulcan Support

Open KodeMunkie opened this issue 3 years ago • 25 comments

I've compiled this on Raspberry PI OS (64 bit) on an RPI 4B 8GB RAM version for use with folding@home. The Pi4 has a Broadcom Videocore 6, which is supported by Vulcan on this OS.

To build I ran: sudo apt install libvulkan-dev

Then followed the clvk instructions here: https://github.com/kpet/clvk#building

This took about 3 hours but produced a libOpenCL.so.

I added the lib to the LD_LIBRARY_PATH and started Folding at home, FAHClient, and an output line that previously said libOpenCL.so couldn't be found was no longer present, but unfortunately no GPUs were listed as a folding "slot".

Assuming a fault in FAHClient I attempted to use libOpenCL.so with xmrig (an open source miner) with the flag --opencl. It was unable to start, so I debugged the xmrig source and found that no GPU devices were detected in a call to clGetDeviceIDs(CL_DEVICE_TYPE_GPU...) changing this to CL_DEVICE_TYPE_DEFAULT was a bit more successful in that I received this output from xmrig:

  • OPENCL #0 (null)/(null)
  • OPENCL GPU #0 n/a llvmpipe (LLVM 11.0.1, 128 bits) 0 MHz cu:1 mem:128/512 MB

But again, it couldn't use the GPU (nb. the memory would also be too low for mining although I had set the GPU memory split to 4096MB "gpu_mem=4096" in the Pi's /boot/config.txt file, I suspect there's a hard limit or something hardcoded somewhere).

EDIT: I've had a look at, and have attached the output from vulkaninfo. vulkan-info.txt

It seems Vulkan is recognising the Broadcom as a CPU type, not GPU (I guess as a system on chip this is probably why).

Can you please advise as to whether it's possible to get clvk working correctly in this scenario - perhaps have clvk override the Vulkan identification of the SoC GPU as a CPU and treat it as a GPU instead (or is this a problem for the Vulkan Broadcom implementers)?

KodeMunkie avatar Jan 15 '22 03:01 KodeMunkie

Can you try running clinfo? I suspect that the call to clGetDevicesIDs in xmrig is not looking for every devices available.

rjodinchr avatar Jan 17 '22 08:01 rjodinchr

I can't get clinfo to build from source

pi@whoyagonnacall:~/src $ cd cl
clinfo/ clvk/
pi@whoyagonnacall:~/src $ cd clinfo/
pi@whoyagonnacall:~/src/clinfo $ make
cc -g -pedantic -Werror -std=c99 -Wall -Wextra -I../OpenCL-Headers   -o clinfo c                                                                            linfo.o  -lOpenCL -ldl
/usr/bin/ld: cannot find -lOpenCL
collect2: error: ld returned 1 exit status
make: *** [Makefile:100: clinfo] Error 1
pi@whoyagonnacall:~/src/clinfo $ ls
clinfo.o  fetch-opencl-dev-win.cmd  legalcode.txt  libOpenCL.so  LICENSE  make.cmd  Makefile  Makefile.win  man1  new-version.sh  README.md  src

As well as having run the clvk make install step I also made sure that libOpenCL.so was in the clinfo folder.

I also tried sudo apt install clinfo which installed however I have no faith in that package since it may have required the package maintainer have a working implementation of opencl to build with - and which doesn't yet exist - and thus perhaps why it returns:

Number of platforms                               0
pi@whoyagonnacall:~ $ sudo clinfo -h
clinfo version 3.0.21.02.21

KodeMunkie avatar Jan 17 '22 11:01 KodeMunkie

To know which libOpenCL.so the application is using, you can set LD_DEBUG=libs in your command line. It will print a lot of things containing which lib the application is linking with. If the libOpenCL.so is not the right one, you can set LD_LIBRARY_PATH={path-to-your-libOpenCL.so}:$LD_LIBRARY_PATH, which should be enough to force the application to use the one you want.

Also for multi-line code (or command) in GitHub, you can use triple quote (instead of simple one) ``` to have something easier to read in the end: this is simple quote

This is a multi-line comment using triple quote:
it respects the end of line

rjodinchr avatar Jan 17 '22 12:01 rjodinchr

I think there may be a problem with the make script in clinfo:

pi@whoyagonnacall:~/src/clinfo $ echo $LD_LIBRARY_PATH
/home/pi/src/clinfo
pi@whoyagonnacall:~/src/clinfo $ make LD_DEBUG=libs
cc -g -pedantic -Werror -std=c99 -Wall -Wextra -I../OpenCL-Headers   -o clinfo clinfo.o  -lOpenCL -ldl
       797:     find library=libc.so.6 [0]; searching
       797:      search cache=/etc/ld.so.cache
       797:       trying file=/lib/aarch64-linux-gnu/libc.so.6
       797:
       797:
       797:     calling init: /lib/ld-linux-aarch64.so.1
       797:
       797:
       797:     calling init: /lib/aarch64-linux-gnu/libc.so.6
       797:
       797:
       797:     initialize program: cc
       797:
       797:
       797:     transferring control: cc
       797:
       798:     find library=libc.so.6 [0]; searching
       798:      search cache=/etc/ld.so.cache
       798:       trying file=/lib/aarch64-linux-gnu/libc.so.6
       798:
       798:
       798:     calling init: /lib/ld-linux-aarch64.so.1
       798:
       798:
       798:     calling init: /lib/aarch64-linux-gnu/libc.so.6
       798:
       798:
       798:     initialize program: /usr/lib/gcc/aarch64-linux-gnu/10/collect2
       798:
       798:
       798:     transferring control: /usr/lib/gcc/aarch64-linux-gnu/10/collect2
       798:
       799:     find library=libbfd-2.35.2-system.so [0]; searching
       799:      search cache=/etc/ld.so.cache
       799:       trying file=/lib/aarch64-linux-gnu/libbfd-2.35.2-system.so
       799:
       799:     find library=libctf.so.0 [0]; searching
       799:      search cache=/etc/ld.so.cache
       799:       trying file=/lib/aarch64-linux-gnu/libctf.so.0
       799:
       799:     find library=libdl.so.2 [0]; searching
       799:      search cache=/etc/ld.so.cache
       799:       trying file=/lib/aarch64-linux-gnu/libdl.so.2
       799:
       799:     find library=libc.so.6 [0]; searching
       799:      search cache=/etc/ld.so.cache
       799:       trying file=/lib/aarch64-linux-gnu/libc.so.6
       799:
       799:     find library=libz.so.1 [0]; searching
       799:      search cache=/etc/ld.so.cache
       799:       trying file=/lib/aarch64-linux-gnu/libz.so.1
       799:
       799:
       799:     calling init: /lib/ld-linux-aarch64.so.1
       799:
       799:
       799:     calling init: /lib/aarch64-linux-gnu/libc.so.6
       799:
       799:
       799:     calling init: /lib/aarch64-linux-gnu/libz.so.1
       799:
       799:
       799:     calling init: /lib/aarch64-linux-gnu/libdl.so.2
       799:
       799:
       799:     calling init: /lib/aarch64-linux-gnu/libbfd-2.35.2-system.so
       799:
       799:
       799:     calling init: /lib/aarch64-linux-gnu/libctf.so.0
       799:
       799:
       799:     initialize program: /usr/bin/ld
       799:
       799:
       799:     transferring control: /usr/bin/ld
       799:
       799:
       799:     calling init: /usr/lib/gcc/aarch64-linux-gnu/10/liblto_plugin.so
       799:
/usr/bin/ld: cannot find -lOpenCL
       799:
       799:     calling fini: /usr/lib/gcc/aarch64-linux-gnu/10/liblto_plugin.so [0]
       799:
       799:
       799:     calling fini: /usr/bin/ld [0]
       799:
       799:
       799:     calling fini: /lib/aarch64-linux-gnu/libctf.so.0 [0]
       799:
       799:
       799:     calling fini: /lib/aarch64-linux-gnu/libbfd-2.35.2-system.so [0]
       799:
       799:
       799:     calling fini: /lib/aarch64-linux-gnu/libdl.so.2 [0]
       799:
       799:
       799:     calling fini: /lib/aarch64-linux-gnu/libz.so.1 [0]
       799:
collect2: error: ld returned 1 exit status
       798:
       798:     calling fini: /usr/lib/gcc/aarch64-linux-gnu/10/collect2 [0]
       798:
       797:
       797:     calling fini: cc [0]
       797:
make: *** [Makefile:100: clinfo] Error 1

KodeMunkie avatar Jan 17 '22 16:01 KodeMunkie

You can try to compile clinfo by yourself, but it will be easier to use the one from apt Everything I explained in the previous message was about executing clinfo, not trying to build it.

rjodinchr avatar Jan 17 '22 16:01 rjodinchr

I tried adding LD_DEBUG=libs as an env variable but it didn't work - there was no change in the clinfo output.

I tried the clinfo from apt which said "Number of platforms 0" (see comment https://github.com/kpet/clvk/issues/337#issuecomment-1014417666 ) so I'm not sure it's working correctly. vulkaninfo reports the device (incorrectly as a CPU but it's a SoC CPU+GPU), xmrig seems to partially reflect the info from the lib i.e. a device exists, and as far as I know there isn't another libOpenCL.so on the system. At this point I don't have enough skill to debug/diagnose where the problem is fully (I only do C occasionally) so I'm happy to shelve this issue and close it :/

KodeMunkie avatar Jan 17 '22 17:01 KodeMunkie

Hi @KodeMunkie and thanks for your interest in clvk.

You're not the first one to try to get FAH going (see https://github.com/kpet/clvk/issues/297). The first thing I suggest you check is whether clvk is correctly detecting the GPU Vulkan implementation. clvk comes with a simple test binary that you can run from the build tree:

$ ./build/simple_test

Is that working? If not, then this means clvk, for some reason, isn't able to use Vulkan on the board. We can dive into this. If yes, then the next step is to get either FAH or xmrig to pick up clvk. Did you use absolute paths when setting LD_LIBRARY_PATH?

kpet avatar Jan 18 '22 18:01 kpet

I didn't see ./build/simple_test in the build tree - I've cleaned and run the build again in case I'd accidentally deleted it. Will report back in a few hours...

KodeMunkie avatar Jan 21 '22 18:01 KodeMunkie

@kpet I finally got the build sorted but didn't see any simple_test:

clspv                  CPackConfig.cmake        lib               llvm-spirv        SPIRV-ToolsConfig.cmake         src
CMakeCache.txt         CPackSourceConfig.cmake  libOpenCL.so      LLVMSPIRVLib.pc   SPIRV-Tools-linkConfig.cmake    tests
CMakeFiles             _deps                    libOpenCL.so.0.1  Makefile          SPIRV-Tools-lintConfig.cmake
cmake_install.cmake    external                 libOpenCL.so.1    sha1_tests        SPIRV-Tools-optConfig.cmake
compile_commands.json  install_manifest.txt     libvulkan.so      SPIRV-Headers.pc  SPIRV-Tools-reduceConfig.cmake

Is there a separate build command for it?

KodeMunkie avatar Jan 26 '22 10:01 KodeMunkie

First, you need to have properly configure cmake: https://github.com/kpet/clvk/blob/master/README.md#tests Then just building clvk should build the test, but you can also force it with this command: cmake --build <path_to_your_build_folder> --target simple_test

rjodinchr avatar Jan 26 '22 10:01 rjodinchr

I did this, passed -DCLVK_BUILD_CONFORMANCE_TESTS=ON , specified the target, and still no build/simple_test was created. The source I've used was checked out and deps pulled in exactly as stated on the github clvk front page.

pi@whoyagonnacall:~/src/clvk/build $ ls -la
total 2952668
drwxr-xr-x 8 pi   pi         4096 Jan 26 10:29 .
drwxr-xr-x 8 pi   pi         4096 Jan 14 20:04 ..
-rwxr-xr-x 1 pi   pi   2344150272 Jan 22 20:50 clspv
-rw-r--r-- 1 pi   pi        91883 Jan 26 10:28 CMakeCache.txt
drwxr-xr-x 5 pi   pi         4096 Jan 26 10:33 CMakeFiles
-rw-r--r-- 1 pi   pi         1782 Jan 21 18:05 cmake_install.cmake
-rw-r--r-- 1 pi   pi      4973986 Jan 26 10:29 compile_commands.json
-rw-r--r-- 1 pi   pi         3410 Jan 21 18:06 CPackConfig.cmake
-rw-r--r-- 1 pi   pi         3848 Jan 21 18:06 CPackSourceConfig.cmake
drwxr-xr-x 4 pi   pi         4096 Jan 14 20:06 _deps
drwxr-xr-x 5 pi   pi         4096 Jan 14 20:06 external
-rw-r--r-- 1 root root        102 Jan 17 11:13 install_manifest.txt
drwxr-xr-x 3 pi   pi         4096 Jan 14 20:30 lib
lrwxrwxrwx 1 pi   pi           14 Jan 26 10:11 libOpenCL.so -> libOpenCL.so.1
-rwxr-xr-x 1 pi   pi    219948976 Jan 26 10:11 libOpenCL.so.0.1
lrwxrwxrwx 1 pi   pi           16 Jan 26 10:11 libOpenCL.so.1 -> libOpenCL.so.0.1
-rwxr-xr-x 1 pi   pi         8208 Jan 14 20:21 libvulkan.so
-rwxr-xr-x 1 pi   pi    424882400 Jan 22 15:59 llvm-spirv
-rw-r--r-- 1 pi   pi          296 Jan 14 20:06 LLVMSPIRVLib.pc
-rw-r--r-- 1 pi   pi      1769794 Jan 26 10:28 Makefile
-rwxr-xr-x 1 pi   pi     27554336 Jan 14 21:05 sha1_tests
-rw-r--r-- 1 pi   pi          172 Jan 14 20:06 SPIRV-Headers.pc
-rw-r--r-- 1 pi   pi          225 Jan 26 10:27 SPIRV-ToolsConfig.cmake
-rw-r--r-- 1 pi   pi          275 Jan 26 10:27 SPIRV-Tools-linkConfig.cmake
-rw-r--r-- 1 pi   pi          275 Jan 26 10:27 SPIRV-Tools-lintConfig.cmake
-rw-r--r-- 1 pi   pi          270 Jan 26 10:27 SPIRV-Tools-optConfig.cmake
-rw-r--r-- 1 pi   pi          285 Jan 26 10:27 SPIRV-Tools-reduceConfig.cmake
drwxr-xr-x 3 pi   pi         4096 Jan 26 10:29 src
drwxr-xr-x 7 pi   pi         4096 Jan 14 20:33 tests
pi@whoyagonnacall:~/src/clvk/build $ cd tests
pi@whoyagonnacall:~/src/clvk/build/tests $ ls
api  CMakeFiles  cmake_install.cmake  Makefile  sha1  simple  simple-from-binary
pi@whoyagonnacall:~/src/clvk/build/tests $ cd simple
pi@whoyagonnacall:~/src/clvk/build/tests/simple $ ls
CMakeFiles  cmake_install.cmake  Makefile

KodeMunkie avatar Jan 26 '22 11:01 KodeMunkie

Did your build complete successfully? It seems some tests were built but not others.

kpet avatar Jan 26 '22 12:01 kpet

It could be due the fact I used make rather than cmake. This is what I get when I try cmake:

pi@whoyagonnacall:~/src/clvk $ cmake --build ~/src/clvk/build/ --target simple_test
gmake: *** No rule to make target 'simple_test'.  Stop.

KodeMunkie avatar Jan 26 '22 12:01 KodeMunkie

-DCLVK_BUILD_CONFORMANCE_TESTS=ON is for the OpenCL CTS to be build, which is not what you are targeting here I believe. Just make sure that you did not set -DCLVK_BUILD_TESTS=OFF (default is to be ON).

The only thing I see that can make the simple_test target not to be produced is if you are setting: -DCLVK_COMPILER_AVAILABLE=OFF during the cmake configuration.

Otherwise, you need to come back to kpet question, did your build complete successfully?

rjodinchr avatar Jan 26 '22 13:01 rjodinchr

It was successful - as far as could see, on first run there were no errors, on the second run unfortunately I don't have the full log from it owing to the terminal session disconnecting, but it did resume at 94% and showed no build issues when I reconnected it and started it again.

I've just tried to run with BUILD TESTS ON:

pi@whoyagonnacall:~/src/clvk $ cmake -DCLVK_BUILD_TESTS=ON --build ~/src/clvk/build/
-- Build type: RelWithDebInfo
-- Did not find googletest, tests will not be built. To enable tests place googletest in '<spirv-dir>/external/googletest'.
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE)
-- Native target architecture is AArch64
-- Threads enabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- Building with -fPIC
-- Targeting AArch64
-- Clang version: 14.0.0
-- Not building amdgpu-arch: hsa-runtime64 not found
-- Registering Bye as a pass plugin (static build: OFF)
-- git version: v0.0.0 normalized to 0.0.0
-- Version: 1.5.4
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Building with -fPIC
-- SPIR-V Headers location is not specified. Will try to download
          spirv.hpp from https://github.com/KhronosGroup/SPIRV-Headers into
          /home/pi/src/clvk/build/external/SPIRV-LLVM-Translator/SPIRV-Headers
-- Building SPIRV-Header examples
-- Installing SPIRV-Header
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
-- Building with -fPIC
-- Found LLVM: 14.0.0git
-- Checking for one of the modules 'SPIRV-Tools'
-- SPIRV-Tools not found; project will be built without --spirv-tools-dis support.
CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:28 (message):
  spirv-as not found! SPIR-V assembly tests will not be run.


CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:33 (message):
  spirv-link not found! SPIR-V test involving the linker will not be run.


CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:38 (message):
  spirv-val not found! SPIR-V generated for test suite will not be validated.


-- CLVK_VULKAN_IMPLEMENTATION = system
-- Vulkan_INCLUDE_DIRS = '/usr/include'
-- Vulkan_LIBRARIES = '/usr/lib/aarch64-linux-gnu/libvulkan.so'
CMake Error at tests/conformance/CMakeLists.txt:22 (add_subdirectory):
  add_subdirectory given source "/home/pi/src/clvk/external/OpenCL-CTS" which
  is not an existing directory.


-- Configuring incomplete, errors occurred!
See also "/home/pi/src/clvk/build/CMakeFiles/CMakeOutput.log".
See also "/home/pi/src/clvk/build/CMakeFiles/CMakeError.log".

I've uploaded these here: CMakeError.log CMakeOutput.log

KodeMunkie avatar Jan 26 '22 14:01 KodeMunkie

cmake has a cache: CMakeCache.txt. Because you did not clean your build repository, the DCLVK_BUILD_CONFORMANCE_TESTS is still to ON, which produce the call to cmake to end with the Configuring incomplete message. Rerun cmake with DCLVK_BUILD_CONFORMANCE_TESTS=OFF. If it complete well, the simple_test target should exist, Otherwise, please attach the logs, and the cache file.

rjodinchr avatar Jan 26 '22 14:01 rjodinchr

After deleting the two log files I ran the following with the conf tests off.

pi@whoyagonnacall:~/src/clvk $ cmake -DCLVK_BUILD_TESTS=ON DCLVK_BUILD_CONFORMANCE_TESTS=OFF --build ~/src/clvk/build/
-- Build type: RelWithDebInfo
-- Did not find googletest, tests will not be built. To enable tests place googletest in '<spirv-dir>/external/googletest'.
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE)
-- Native target architecture is AArch64
-- Threads enabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- Building with -fPIC
-- Targeting AArch64
-- Clang version: 14.0.0
-- Not building amdgpu-arch: hsa-runtime64 not found
-- Registering Bye as a pass plugin (static build: OFF)
-- git version: v0.0.0 normalized to 0.0.0
-- Version: 1.5.4
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Building with -fPIC
-- SPIR-V Headers location is not specified. Will try to download
          spirv.hpp from https://github.com/KhronosGroup/SPIRV-Headers into
          /home/pi/src/clvk/build/external/SPIRV-LLVM-Translator/SPIRV-Headers
-- Building SPIRV-Header examples
-- Installing SPIRV-Header
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
-- Building with -fPIC
-- Found LLVM: 14.0.0git
-- Checking for one of the modules 'SPIRV-Tools'
-- SPIRV-Tools not found; project will be built without --spirv-tools-dis support.
CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:28 (message):
  spirv-as not found! SPIR-V assembly tests will not be run.


CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:33 (message):
  spirv-link not found! SPIR-V test involving the linker will not be run.


CMake Warning at external/SPIRV-LLVM-Translator/test/CMakeLists.txt:38 (message):
  spirv-val not found! SPIR-V generated for test suite will not be validated.


-- CLVK_VULKAN_IMPLEMENTATION = system
-- Vulkan_INCLUDE_DIRS = '/usr/include'
-- Vulkan_LIBRARIES = '/usr/lib/aarch64-linux-gnu/libvulkan.so'
CMake Error at tests/conformance/CMakeLists.txt:22 (add_subdirectory):
  add_subdirectory given source "/home/pi/src/clvk/external/OpenCL-CTS" which
  is not an existing directory.


-- Configuring incomplete, errors occurred!

No error files were output. I also tried adding an empty folder in the location it requested and running again but same problem - i.e. no make file in that folder.

KodeMunkie avatar Jan 26 '22 17:01 KodeMunkie

Sorry, I forget the - before DCLVK_BUILD_CONFORMANCE_TESTS=OFF, so it did not change anything. Please retry with -DCLVK_BUILD_CONFORMANCE_TESTS=OFF

rjodinchr avatar Jan 26 '22 17:01 rjodinchr

Build was successful.

pi@whoyagonnacall:~/src/clvk/build $ ./simple_test
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Platform: clvk
Device: llvmpipe (LLVM 11.0.1, 128 bits)
Buffer content verified, test passed.

KodeMunkie avatar Jan 26 '22 17:01 KodeMunkie

So this looks like it's using a software implementation of vulkan instead of the hardware driver. I'll do some digging as to why. (At least I think it is - I'm out of my depth here).

KodeMunkie avatar Jan 26 '22 18:01 KodeMunkie

Partial success - vulkaninfo reports a different set of devices/layers when not run headless:

Layers: count = 2
=================
VK_LAYER_MESA_device_select (Linux device selection layer) Vulkan version 1.2.73, layer version 1:
	Layer Extensions: count = 0
	Devices: count = 2
		GPU id = 0 (V3D 4.2)
		Layer-Device Extensions: count = 0

		GPU id = 1 (llvmpipe (LLVM 11.0.1, 128 bits))
		Layer-Device Extensions: count = 0

libOpenCL however, according to xmrig's use of the getDeviceIds/getDeviceInfo, is only returning GPU id 1 (afaics, libOpenCL is not returning all devices in xmrig's OclLib.cpp class - i.e. not an xmrig fault).

Could there be a similar issue with clvk also querying hardware, but always in headless mode, seeing only 1 device/layer, regardless of a started front end?

KodeMunkie avatar Jan 27 '22 10:01 KodeMunkie

To contrast those 2 layers with simple_test's output now (which hasn't changed to reflect the V3D 4.2 GPU)

WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Platform: clvk
Device: llvmpipe (LLVM 11.0.1, 128 bits)
Buffer content verified, test passed.

I suspect there is a bug in clvk's device.cpp but would like to confirm where it is dropping the V3D device.

KodeMunkie avatar Jan 27 '22 12:01 KodeMunkie

Bingo!

[CLVK] Initialising device V3D 4.2
[CLVK]   API Version: 1.0.155
[CLVK] Physical device (Integrated GPU) has 1 queue families:
[CLVK]   queue family 0:  1 queues | GRAPHICS COMPUTE TRANSFER 
...
[CLVK] loader_validate_device_extensions: Device extension VK_KHR_storage_buffer_storage_class not supported by selected physical device or enabled layers.
[CLVK] vkCreateDevice:  Failed to validate extensions in list
[CLVK] Failed to create a device : VK_ERROR_EXTENSION_NOT_PRESENT
[CLVK] Could not create CL device from Vulkan device!

Full log info here: fulllog.txt

KodeMunkie avatar Jan 27 '22 14:01 KodeMunkie

I attach a patch that prevents the device extension killing the loader. Unfortunately as it's a dirty hack it removes an extension altogether but I include it here to illustrate the issue for others. I have no idea what the extension did but now OpenCL appears to work and the V3D device is being recognised by simple test and xmrig.

pi@whoyagonnacall:~/work/clvk $ ./build/simple_test
WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Platform: clvk
Device: V3D 4.2
Buffer content verified, test passed.

changes.txt

KodeMunkie avatar Jan 27 '22 20:01 KodeMunkie

Bingo!

[CLVK] Initialising device V3D 4.2
[CLVK]   API Version: 1.0.155
[CLVK] Physical device (Integrated GPU) has 1 queue families:
[CLVK]   queue family 0:  1 queues | GRAPHICS COMPUTE TRANSFER 
...
[CLVK] loader_validate_device_extensions: Device extension VK_KHR_storage_buffer_storage_class not supported by selected physical device or enabled layers.
[CLVK] vkCreateDevice:  Failed to validate extensions in list
[CLVK] Failed to create a device : VK_ERROR_EXTENSION_NOT_PRESENT
[CLVK] Could not create CL device from Vulkan device!

Full log info here: fulllog.txt

what are you running here? simple_test?

tim@tim-desktop:~/clvk/build$ ./simple_test [CLVK] Unrecognized device 'llvmpipe (LLVM 12.0.1, 128 bits)', some device properties will be incorrect. WARNING: lavapipe is not a conformant vulkan implementation, testing use only. [CLVK] Unrecognized device 'V3D 4.2', some device properties will be incorrect. [CLVK] The VK_KHR_driver_properties extension is not supported. [CLVK] Using default Vulkan driver behaviors. [CLVK] This device does not support VK_EXT_calibrated_timestamps or it does not support the required VK_TIME_DOMAIN_CLOCK_MONOTONIC_EXT and VK_TIME_DOMAIN_DEVICE_EXT time domains [CLVK] clGetHostTimer and clGetDeviceAndHostTimer will not work [CLVK] Command queue profiling will suffer from limitations Platform: clvk Device: llvmpipe (LLVM 12.0.1, 128 bits) [CLVK] compile_source: failed to compile the program

alexander-toschev avatar Apr 15 '22 10:04 alexander-toschev