Raspberry Pi Issues

[geefr]

Did a quick test on a Pi 4, latest Raspbian image. Built on device, against system packages (had to install vulkan-dev, mesa-vulkan-drivers, etc).

Compiling

VSG itself compiles, vsgExamples / exchange executables raise errors on libatomic.

Fix should be a cmake update, with a check whether atomic is built-in. Adding -latomic to the vsg.a link resolves it for vsgExamples.

Examples

Ran a few examples, basic rendering seems fine, input works, no obvious quirks.

vsgtext, vsgclip, vsgtexturearray segfault, here's the output:

WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only.
VUID_Undefined(ERROR / SPEC): msgNum: 2044605652 - Validation Error: [ VUID_Undefined ] Object 0: handle = 0x15293a0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x79de34d4 | vkCreateShaderModule: parameter pCreateInfo->codeSize / 4 must be greater than 0.
    Objects: 1
        [0] 0x15293a0, type: 3, name: NULL
UNASSIGNED-CoreValidation-Shader-InconsistentSpirv(ERROR / SPEC): msgNum: 7060244 - Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x15293a0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: Invalid SPIR-V magic number.
    Objects: 1
        [0] 0x15293a0, type: 3, name: NULL

[robertosfield]

I'm thinking that the -latomic addition could be added by the VulkanSceneGraph/src/vsg/CMakeLists.txt as part of the LIBRARIES list in much the same way that Vulkan and Xcb are added right now. We'd need code detecting the build on Raspbian, or perhaps use it for all ARM builds.

The SPIR-V problem may well be a driver issue as vsgtext, vsgclip and vsgtexturearray all run cleanly when you run with the Vulkan debug layer enabled using the -d command line options i.e.

vsgtext -d

So initially I think we should try to get the VSG, vsgXchange + vsgExamples all building cleanly on Pi 4, then keep checking the updates to the drivers to see if they resolve the error report. On this theme, how often do new Vulkan drivers get released?

[geefr]

Yes, atomic as part of LIBRARIES is good. Perhaps this is a suitable example: https://github.com/llvm-mirror/llvm/blob/master/cmake/modules/CheckAtomic.cmake

It's possible needing -latomic is more of a compiler issue than specifically being Raspbain/ARM - Quite possible the same thing is present on Android, maybe desktop clang builds? My test build was using GCC 10.2.1.

Drivers: Not entirely sure but for the Pi 4 it's the V3DV driver in mesa. Under active development by Igalia, currently somewhere between Vulkan 1.1 and 1.2. Looks like there's some Vulkan support for older Pis, but I'll ignore that for now.

As it's mesa I'd expect regular updates through the package manager, past updates have been on cycles of around 1-6 months. last update was 5th May, and there are active changes related to SPIR-V going in.

Could certainly test with un-compiled vsgt? Testing the latest mesa dev would be possible too, it's not too tricky to build.

[geefr]

Re-tested with a fresh install / pi, realised the last build I did was on the 32-bit version of Raspbian (backwards compatible with the older pis).

On the 64-bit build the only difference is the '-latomic' wasn't needed, compiled out of the box. No apparent change to running the samples, build was on same hardware, and gcc 10.2.1 again.

[robertosfield]

On Mon, 30 May 2022 at 17:17, Gareth Francis @.***> wrote:

Re-tested with a fresh install / pi, realised the last build I did was on the 32-bit version of Raspbian (backwards compatible with the older pis).

On the 64-bit build the only difference is the '-latomic' wasn't needed, compiled out of the box. No apparent change to running the samples, build was on same hardware, and gcc 10.2.1 again.

Does this mean the 32bit build doesn't have processor support for atomics?

Is there Cmake support for detecting 32bit vs 64bit Rasbian builds?

I'm just wondering if we can automate the -latomic addition. If not then a note on INSTALL.md would be appropriate.

Message ID: @.***>

[geefr]

Processor support: Maybe, it is armv7l vs aarch64 but haven't dug into the specifics, think it's also tied into how gcc was built.

Detecting it in CMake: Took a little fiddling but yes, think that should do it.

[robertosfield]

@geefr Is the Raspberry Pi4 support now solid enough to close this Issue?

[geefr]

Building + linking + basic usage yes

Anything that loads shaders - not sure if the vulkan driver is ready yet, but perhaps that's a separate issue. Looks like the crashes are when it's translating SPIR-V to whatever the GPU's instruction set is..

I did try a slightly newer mesa but no luck on the crashes, the error did change though. Next steps would be to get the latest mesa building, but I couldn't manage that on the stock raspbian release

[robertosfield]

On Wed, 29 Jun 2022 at 19:30, Gareth Francis @.***> wrote:

Building + linking + basic usage yes

Anything that loads shaders - not sure if the vulkan driver is ready yet, but perhaps that's a separate issue. Looks like the crashes are when it's translating SPIR-V to whatever the GPU's instruction set is..

I did try a slightly newer mesa but no luck on the crashes, the error did change though. Next steps would be to get the latest mesa building, but I couldn't manage that on the stock raspbian release

Are you linking against glslang? Do the problem models/application required GLSL -> SPRI-V compilation?

One of things I want to do before 1.0 is provide better feedback on when GLSL shaders can't be compiled to SPIR-V as right now it crashes. It won't be complicated to do, just been one item near the back of my TODO list.

Message ID: @.***>

[geefr]

I think I'm linking glslang yes, but haven't checked that build in a while.

The crashes were with the vsgExamples models, which I think are already SPIR-V? Definitely it's a segfault within the graphics driver, rather than a simple exception / vsg level thing. (I've got some notes/traces on one of the pis, might have time over the weekend to add them here)

[robertosfield]

Do any of the VSG example run?

I recall users previously having quite large models rendering on the Pi, so it feels like a regression.

[geefr]

Yes, many samples run, basic rendeirng of the teapot, lz.vsgt, etc is fine. The ones that don't hit segfaults in the driver, or other similar failures.

Using vsgbuilder as a test (similar case with vsgtext and vsgstatecomposer)

With mesa 20.3.5 (default on latest 64-bit raspbian release):

WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only.

Thread 1 "vsgbuilder" received signal SIGSEGV, Segmentation fault.
0x0000007ff65d4c44 in ?? () from /usr/lib/aarch64-linux-gnu/libvulkan_broadcom.so
(gdb) bt
#0  0x0000007ff65d4c44 in ?? () from /usr/lib/aarch64-linux-gnu/libvulkan_broadcom.so
#1  0x0000007ff6563bd8 in ?? () from /usr/lib/aarch64-linux-gnu/libvulkan_broadcom.so
#2  0x0000007ff6566bf4 in ?? () from /usr/lib/aarch64-linux-gnu/libvulkan_broadcom.so
#3  0x000000555570ea40 in vsg::GraphicsPipeline::Implementation::Implementation (this=0x7ff7949860, context=..., device=
    0x7ff7947f10, renderPass=0x7ff7948118, pipelineLayout=0x7ff79461c8, 
    shaderStages=std::vector of length 2, capacity 2 = {...}, 
    pipelineStates=std::vector of length 9, capacity 9 = {...}, subpass=0)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/state/GraphicsPipeline.cpp:156
#4  0x000000555570f5d8 in vsg::Inherit<vsg::Object, vsg::GraphicsPipeline::Implementation>::create<vsg::Context&, vsg::ref_ptr<vsg::Device>&, vsg::ref_ptr<vsg::RenderPass>&, vsg::ref_ptr<vsg::PipelineLayout>&, std::vector<vsg::ref_ptr<vsg::ShaderStage>, std::allocator<vsg::ref_ptr<vsg::ShaderStage> > >&, std::vector<vsg::ref_ptr<vsg::GraphicsPipelineState>, std::allocator<vsg::ref_ptr<vsg::GraphicsPipelineState> > >&, unsigned int&> ()
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/core/Inherit.h:38
#5  0x000000555570e734 in vsg::GraphicsPipeline::compile (this=0x7ff79463d0, context=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/state/GraphicsPipeline.cpp:120
#6  0x000000555570ee34 in vsg::BindGraphicsPipeline::compile (this=0x7ff7844040, context=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/state/GraphicsPipeline.cpp:216
#7  0x00000055556f28d4 in vsg::StateGroup::compile (this=0x7ff78440b0, context=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/nodes/StateGroup.cpp:59
#8  0x00000055557477e8 in vsg::CompileTraversal::apply (this=0x7ff7948880, stateGroup=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/traversals/CompileTraversal.cpp:208
#9  0x00000055556f37ac in vsg::Inherit<vsg::Group, vsg::StateGroup>::accept (this=0x7ff78440b0, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/core/Inherit.h:71
#10 0x00000055555ff9f0 in vsg::Group::t_traverse<vsg::Group, vsg::Visitor> (node=..., visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:36
#11 0x00000055555ff8bc in vsg::Group::traverse (this=0x7ff7844010, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:39
#12 0x0000005555747624 in vsg::CompileTraversal::apply (this=0x7ff7948880, object=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/traversals/CompileTraversal.cpp:185
#13 0x00000055555f6334 in vsg::Visitor::apply (this=0x7ff7948880, value=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/core/Visitor.cpp:422
#14 0x00000055555f63ac in vsg::Visitor::apply (this=0x7ff7948880, value=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/core/Visitor.cpp:430
#15 0x0000005555600c94 in vsg::Inherit<vsg::Node, vsg::Group>::accept (this=0x7ff7844010, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/core/Inherit.h:71
#16 0x00000055555ff9f0 in vsg::Group::t_traverse<vsg::Group, vsg::Visitor> (node=..., visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:36
#17 0x00000055555ff8bc in vsg::Group::traverse (this=0x7ff7844a28, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:39
#18 0x0000005555748000 in vsg::CompileTraversal::apply (this=0x7ff7948880, view=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/traversals/CompileTraversal.cpp:301
#19 0x0000005555671e84 in vsg::View::t_accept<vsg::View, vsg::Visitor> (node=..., visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/viewer/View.h:40
#20 0x0000005555671ce8 in vsg::View::accept (this=0x7ff7844a28, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/viewer/View.h:45
#21 0x00000055555ff9f0 in vsg::Group::t_traverse<vsg::Group, vsg::Visitor> (node=..., visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:36
#22 0x00000055555ff8bc in vsg::Group::traverse (this=0x7ff7844c00, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:39
#23 0x0000005555747ddc in vsg::CompileTraversal::apply (this=0x7ff7948880, renderGraph=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/traversals/CompileTraversal.cpp:281
#24 0x0000005555668774 in vsg::Inherit<vsg::Group, vsg::RenderGraph>::accept (this=0x7ff7844c00, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/core/Inherit.h:71
#25 0x00000055555ff9f0 in vsg::Group::t_traverse<vsg::Group, vsg::Visitor> (node=..., visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:36
#26 0x00000055555ff8bc in vsg::Group::traverse (this=0x7ff7844988, visitor=...)
--Type <RET> for more, q to quit, c to continue without paging--
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/nodes/Group.h:39
#27 0x0000005555747aa8 in vsg::CompileTraversal::apply (this=0x7ff7948880, commandGraph=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/traversals/CompileTraversal.cpp:242
#28 0x0000005555665524 in vsg::Inherit<vsg::Group, vsg::CommandGraph>::accept (this=0x7ff7844988, visitor=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/include/vsg/core/Inherit.h:71
#29 0x0000005555631a00 in vsg::Viewer::compile (this=0x7ff7946fc8, hints=...)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg/viewer/Viewer.cpp:303
#30 0x00000055555d6158 in main (argc=1, argv=0x7ffffff038)
    at /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/vsgExamples/examples/utils/vsgbuilder/vsgbuilder.cpp:253

And with mesa 21.3.9 it's at least an assertion instead of a segfault:

Thread 1 "vsgbuilder" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000007ff7ae8ea0 in __GI_abort () at abort.c:79
#2  0x0000007ff7af5878 in __assert_fail_base (fmt=0x7ff7bf32b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x7ff64f9dc8 "nir", file=file@entry=0x7ff64f9c50 "../src/broadcom/vulkan/v3dv_pipeline.c", 
    line=line@entry=454, function=function@entry=0x7ff64fbca0 "shader_module_compile_to_nir") at assert.c:92
#3  0x0000007ff7af58dc in __GI___assert_fail (assertion=0x7ff64f9dc8 "nir", 
    file=0x7ff64f9c50 "../src/broadcom/vulkan/v3dv_pipeline.c", line=454, 
    function=0x7ff64fbca0 "shader_module_compile_to_nir") at assert.c:101
#4  0x0000007ff63212f8 in ?? ()

There are newer mesa versions available, but I couldn't build them - I'd need a newer libdrm and similar dependencies, haven't had time to dig into that yet.

And my cmake config for reference - yes glslang is present, there don't seem to be any obvious exceptions / vulkan errors, besides the driver crashes.

 GenericCodeGen_LIBRARY           GenericCodeGen_LIBRARY-NOTFOUND
 HLSL_LIBRARY                     HLSL_LIBRARY-NOTFOUND
 MAINTAINER                       OFF
 MachineIndependent_LIBRARY       MachineIndependent_LIBRARY-NOTFOUND
 OGLCompiler_LIBRARY              OGLCompiler_LIBRARY-NOTFOUND
 OSDependent_LIBRARY              OSDependent_LIBRARY-NOTFOUND
 SPIRV-Tools-opt_LIBRARY          /usr/lib/aarch64-linux-gnu/libSPIRV-Tools-opt.a
 SPIRV-Tools_LIBRARY              /usr/lib/aarch64-linux-gnu/libSPIRV-Tools.a
 SPIRV_LIBRARY                    SPIRV_LIBRARY-NOTFOUND
 VSG_MAX_DEVICES                  1
 VSG_WARNING_FLAGS                -Wall;-Wparentheses;-Wno-long-long;-Wno-import;-Wreturn-type;-Wmissing-braces;-Wunknown
 spirv_INCLUDE_DIR                spirv_INCLUDE_DIR-NOTFOUND
 vsg_glslang_DIR                  /media/gareth/a808383f-032d-4c03-92c2-2d91361f3d091/vsg/VulkanSceneGraph/src/vsg

[geefr]

These all seem to function without any unexpected issues:

• vsgallocator
• vsganaglyphicstereo
• vsgarrays
• vsgcameras
• vsggroups
• vsgintersection
• vsgmeshshader
  • Mesh shaders not supported
• vsgio
• vsglights
• vsgmaths
• vsgmemory
• vsgmultigpu
• vsgmultiviews
• vsgortho
• vsgoverlay
• vsgpath
• vsgpointer
• vsgtypes
• vsgvalues
• vsgviewer
• vsgvisitor
• vsgvisitorcustomtype
• vsgwindows
  • Input to 2nd window goes to 1st one instead? Desn't seem pi-specific
• vsgraytracing
  • Fails to create device, as expected
• vsgcompute
• vsgdraw
• vsginterleaved
• vsgstateswitch
• vsgrendertotexture
• vsgsubpass
• vsgtexturearray

[robertosfield]

Thanks for the testing and info.

It might be that not enough of glslang is available so the ShaderCompiler isn't be compiled against glslang. With VSG master I've added some extra checks and message that shader compilation isn't available.

I will compile the VSG without glslang and see if I get the same error as you with vsgbuilder. I also need to add some extra reporting/better handling of what shaders aren't compiled and are needed.

Which version/date of the VSG didn't you use for these tests? Are you using VulkanSDK or locally installed vulkan, glsang and associated headers?

[geefr]

Latest vsg master as of this morning, vulkan glslang from the system, nothing special there.

From the assert and various mesa bug reports I suspect it's something in the driver itself, but hadn't tried any other vulkan samples, or the newer mesa yet. There's definitely active development on the v3Dv driver, and a lot of changes since the version used by raspbian..

[robertosfield]

RaspberryPi have met Vulkan-1.2 conformance checks so could things be working better now?

https://www.raspberrypi.com/news/vulkan-update-version-1-2-conformance-for-raspberry-pi-4/

[geefr]

Yep noted that. The slightly newer mesa version I tried previously did change the error messages, so I'm taking that as a positive sign.

Haven't booted up the pi yet but a note here indicates the vulkan 1.2 update may not reach the package manager until end of year: https://www.theregister.com/2022/08/03/pi_4_vulkan/ (I'm sure somewhere will have upgrade instructions before that though. Will test if I find the time)

[geefr]

Managed to build mesa 22.2.0 for the pi and yes - Resolves the segfaults as far as I can tell, things seem to run well and with decent performance.

So I think that means the pi 4 can be called working, once the mesa 22 update reaches the package manager / everyone.

[robertosfield]

On Sat, 14 May 2022 at 21:03, Gareth Francis @.***> wrote:

Did a quick test on a Pi 4, latest Raspbian image. Built on device, against system packages (had to install vulkan-dev, mesa-vulkan-drivers, etc). Compiling

VSG itself compiles, vsgExamples / exchange executables raise errors on libatomic.

Fix should be a cmake update, with a check whether atomic is built-in. Adding -latomic to the vsg.a link resolves it for vsgExamples.

Could you generate a PR for this fix?

Examples

Ran a few examples, basic rendering seems fine, input works, no obvious quirks.

vsgtext, vsgclip, vsgtexturearray segfault, here's the output:

WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only. VUID_Undefined(ERROR / SPEC): msgNum: 2044605652 - Validation Error: [ VUID_Undefined ] Object 0: handle = 0x15293a0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x79de34d4 | vkCreateShaderModule: parameter pCreateInfo->codeSize / 4 must be greater than 0. Objects: 1 [0] 0x15293a0, type: 3, name: NULL UNASSIGNED-CoreValidation-Shader-InconsistentSpirv(ERROR / SPEC): msgNum: 7060244 - Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x15293a0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: Invalid SPIR-V magic number. Objects: 1 [0] 0x15293a0, type: 3, name: NULL

Running these examples on my Kubuntu 21.10 with AMD 5700G with -d doesn't report any issues.

This doesn't rule out an error with SPIR-V generation. In the case of vsgtext and vsgclip they will be generating the SPIR-V at runtime using GLslang, or at least try to.

Do you have GLslang installed and is the VSG built against it?

Message ID: @.***>