Various crashes on Arch Linux with Wayland

[e-dant]

Thoughts on the Issue

Filament's copy of SDL seems unable to produce a valid SDL environment. Moreover, the most recent version of SDL, when used in place of Filament's copy of SDL, also fails to produce a valid SDL environment when running Filament's (desktop) samples.

This is likely not an issue with SDL. Programs which I have developed using SDL run fine on Arch. (However, I have not been able to find a "standard SDL test" program out there to see if it will work.) This leads me to believe that there is some way in which Filament is not handling SDL properly, causing crashes on (Arch) Linux with Wayland.

Steps to reproduce

While using a freshly installed or otherwise containerized distribution of Arch Linux with Wayland as the compositor and a Wayland-capable Window Manager such as sway, install dependencies for Filament (sudo pacman -Sy sway wayland wayland-protocols sdl2 clang ninja cmake) and compile filament from source (export CC=[clang binary]; export CXX=[clang++binary], etc, ./build.sh release).

Then launch any of the sample programs (such as ./out/cmake-[release|debug]/samples/hellotriangle)

The program will crash.

Relevant output from an unmodified Filament source at version 1.19.1

[1319/1319] Linking CXX executable samples/rendertarget
$ ./out/cmake-release/samples/hellotriangle
[1]    14755 segmentation fault (core dumped)  ./out/cmake-release/samples/hellotriangle
$ ./out/cmake-release/samples/suzanne
[1]    14792 segmentation fault (core dumped)  ./out/cmake-release/samples/suzanne

Note: The same output occurs with any of the following coditions:

• The system is using XWayland with the FILAMENT_SUPPORTS_WAYLAND set to OFF
• The system is using Wayland (not X or XWayland) with FILAMENT_SUPPORTS_WAYLAND set to OFF
• The system is using Wayland with FILAMENT_SUPPORTS_WAYLAND set to ON
• (Described in the next section) different output, but a crash nevertheless when the system is using Wayland with FILAMENT_SUPPORTS_WAYLAND set to ON and SDL2 linked against pacman's SDL2 package

Relevant output from modified Filament source at version 1.19.1

Modifications include and are strictly limited to the notes below.

Notes on modifications

Note: Some debugging out was added to libs/filamentapp/src/NativeWindowHelperLinux.cpp as follows:

    SDL_version compiled;
    SDL_version linked;

    SDL_VERSION(&compiled);
    SDL_GetVersion(&linked);
    std::cout << "We compiled against SDL version "<<compiled.major<<"."<<compiled.minor<<"."<<compiled.patch
    std::cout << "We linked against SDL version "<<linked.major<<"."<<linked.minor<<"."<<linked.patch<<"\n";

Note: Changes to CMakeLists.txt (root cmake):

option(FILAMENT_SKIP_SDL2 "Skip dependencies of SDL2, and SDL2" ON)
option(FILAMENT_SUPPORTS_WAYLAND "Include Wayland support in Linux builds" ON)

Note: Changes to CMake to reproduce above in samples/CMakeLists.txt (add_demo function):

function(add_demo NAME)
    find_package(SDL2 REQUIRED)
    include_directories(${SDL2_INCLUDE_DIRS})
    include_directories(${GENERATION_ROOT})

    add_executable(${NAME} ${NAME}.cpp)

    target_link_libraries(${NAME} PRIVATE sample-resources filamentapp ${SDL2_LIBRARIES})
    target_compile_options(${NAME} PRIVATE ${COMPILER_FLAGS})
endfunction()

Output

Here is the Vulkan driver crashing after linking with the most recent version of SDL2 outside of filament (via sudo pacman -Sy sdl2 and adding CMake rules described above):

1 warning generated.
[55/55] Linking CXX executable samples/texturedquad
We compiled against SDL version ..
We linked against SDL version ..
FEngine (64 bits) created at 0x7fd5b8243010 (threading is enabled)
FEngine resolved backend: Vulkan
N5utils18PostconditionPanicE
in filament::backend::VulkanDriver::VulkanDriver(filament::backend::VulkanPlatform *, const char *const *, uint32_t):229
reason: Unable to create Vulkan instance.
#0	hellotriangle                       0x55ae9260a6e0 (null) + 94208268478176
#1	hellotriangle                       0x55ae925785bd (null) + 94208267879869
#2	hellotriangle                       0x55ae9257778e (null) + 94208267876238
#3	hellotriangle                       0x55ae92563fef (null) + 94208267796463
#4	hellotriangle                       0x55ae925096f1 (null) + 94208267425521
#5	hellotriangle                       0x55ae9250fb9e (null) + 94208267451294
#6	libc.so.6                           0x7fd5c48995c2 (null) + 140556102112706


libc++abi: terminating with uncaught exception of type utils::PostconditionPanic
[1]    5594 IOT instruction (core dumped)  ./out/cmake-release/samples/hellotriangle

[e-dant]

Update after testing

SDL2 Tests

To Reproduce

SDL was compiled out-of-directory with CMake.

Samples were compiled from source from SDL's github repo here.

The source was unmodified except for:

• Edited the CMakeLists to enable tests
• The cmake policy for GLVND was set to NEW (in /usr/share/cmake/Modules/FindOpenGL.cmake).

Testing programs run

• testdraw2 (success)
• testgl2 (success)
• testgles2 (success)
• testgeometry (success)
• testdrawchessboard (success)
• testfilesystem (success)
• testshader (success)
• testatomic (success)
• testgles (failure)
• --> testnative (failure) (This may be an interesting failure. See the bold note in the next section: "Results.")

Results

All tests programs were successful except for testnative and testgles. Note that testgles failed but testgles2 succeeded.

The samples sometimes open floating windows, other times open normal windows. I am unsure if this is working exactly as intended, but it seems harmless. The same samples always open the same kinds of windows.

The output from testnative is probably relevant here:

ERROR: Couldn't find native window code for wayland driver

More Filament Testing -> OpenGL-Only

To Reproduce

Follow the steps in the original issue for the modified filament compilation. Then, either:

• Pass Backend::OPENGL in libs/filamentapp/src/FilamentApp.cpp instead of DEFAULT.
• Disable Vulkan from the root CMakeLists.txt

The same segfault occurs in both of the above cases.

Output

./out/cmake-release/samples/hellotriangle
FEngine (64 bits) created at 0x7f9ce0a5a010 (threading is enabled)
FEngine resolved backend: OpenGL
Selected backend not supported in this build.
[1]    7282 segmentation fault (core dumped)  ./out/cmake-release/samples/hellotriangle

Request

Give me some pointers or hunches to where you think this issue might be coming from. I'll fiddle with it and see if I can solve it.

[e-dant]

Closing because dead