[BUG]: Segmentation Fault Error on Linux Server Build with WebRTC-exp7 or above

[ovsky]

Package version

2.4.0-exp.7 or above

Environment

* OS: Ubuntu 20.04.4 LTS
* Unity version: 2021.3.5f1 LTS

Steps To Reproduce

1. Build Unity project with WebRTC-exp7 or above for Linux Dedicated Server Platform.
2. Try to run it on any Linux distribution.

Current Behavior

Loading application causes the Segmentation fault error when loading libwebrtc.so.

[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
Preloaded 'lib_burst_generated.so'
Preloaded 'libwebrtc.so'
Segmentation fault

...and server crashes at this point.

Expected Behavior

The package should be loaded without any problems, like with removed WebRTC library:

    (...)
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
Preloaded 'lib_burst_generated.so'
Initialize engine version: 2021.3.5f1 (40eb3a945986)

Anything else?

I have installed all dependencies and packages that may be needed to make WebRTC working properly. This bug occurs on any build settings (Mono/.NET Framework/.NET Core/Burst and no-Burst). Previoulsy there will be no problem to run

Packages above exp.6 - 7, 8, and 9 causes the same problem.

Interesting fact

After removing the libwebrtc.so package from build's Data>Plugins the server works properly (as shown above) - obviously, without WebRTC functionalities.

[karasusan]

Is it same issue? https://github.com/Unity-Technologies/com.unity.webrtc/issues/753

[ovsky]

No, it's another case, because in my issue the application just won't start - it is unable to load the libwebrtc.so library, so there aren't problems with signaling or streaming for now. But it's a bit similar - beacuse using the older version of the package also worked for the suggested issue reporter as well.

[karasusan]

Can I have more detail of the Segmentation fault? It might put the crash log.

[ovsky]

So, as this is a segmentation error, it may will be best if I leave you a valgrind logs (from Ubuntu WSL this time).

1. from basic app verison: valgrind.log
2. from app with debug mode and script debugging: valgrind-debug.log

The Unity application does not leave any crash logs on its own.

Edit: Ok, I think it won't be super helpful in this situation, but I will also leave the strace logs:

1. strace.log
2. strace-debug.log

[karasusan]

Thank you for sharing the detail. We will check it.

[karasusan]

memo: WRS-388

[ZenBre4ker]

Can confirm, that this only happens on the "Dedicated Server Build".

Libwebrtc.so just can't be loaded and leads to a Segmentation fault.

Still with the newest version 3.0.0.-pre.1 274791c

[ZenBre4ker]

I started the server with LD_DEBUG=all @karasusan Maybe this debug-log of the preload can help. Debug-Libwebrtc.so.txt

It seems to miss references in the libktx_unity.so and libwebrtc.so like:

• UnitySetGraphicsDevice
• UnityRenderEvent
• UnityGetAudioEffectDefinitions

These are only missing in the libktx_unity.so

• UnityPluginLoad
• UnityPluginUnload

[karasusan]

@ZenBre4ker Thank you for the detail. I have a question. The dedicated server build strips the graphic feature from the binary. This app cannot render the graphics. Is this OK for your use-case?

[ZenBre4ker]

@karasusan Yeah that's fine. The server build in my use-case only needs to receive and relay webrtc-streams. In general a headless build doesnt need graphics, I assumed.

[karasusan]

@ZenBre4ker All right. I added the task into our backlog.

[keepingitneil]

Also seeing this for a headless linux build. Adding this comment to help inform priority. Thanks!

[davidyrgilbert]

This is still quite a breaking issue for us, is there any new information regarding the segfault?

[karasusan]

I'm surprised that there are many request for this issue. I'm curious to see the main use case using the headless linux build.

[gtk2k]

Using cloud rendering.

[davidyrgilbert]

• Data streams from the server, which is running headless.
• Authorative server needs to be kept in the audio/video streaming loop as well, even if it's just for control. Could get away with abstracting this away from the plugin, but that makes everything even more convoluted

[karasusan]

@gtk2k I haven't checked the video streaming feature for the dedicated server, does it work?

[gtk2k]

@karasusan It's not headless at the moment, but we're planning to move to headless in the future if it works.

[karasusan]

@gtk2k I commented about the dedicated server, I think it might not work as you expected. https://github.com/Unity-Technologies/com.unity.webrtc/issues/791#issuecomment-1329999315

[ZenBre4ker]

@karasusan Small Update concerning the Segmentation Fault. We made a small test-project only consisting of the webrtc sample and built via Unity UI directly a dedicated Server as linux .x86_64 and as windows .exe. While the dedicated Server for Windows works, stating this:

[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
Initialize engine version: 2021.3.19f1 (c9714fde33b6)
[Subsystems] Discovering subsystems at path D:/Test_Projekt/WebRtc_Sample_Test/Builds/Windows/WebRtc_Sample_Test_Data/UnitySubsystems
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0; jobified=0
NullGfxDevice:
    Version:  NULL 1.0 [1.0]
    Renderer: Null Device
    Vendor:   Unity Technologies
ERROR: Shader Sprites/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
Microsoft Media Foundation video decoding to texture disabled: graphics device is Null, only Direct3D 11 and Direct3D 12 (only on desktop) are supported for hardware-accelerated video decoding.
ERROR: Shader Sprites/Mask shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader GUI/Text Shader shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
UnloadTime: 0.546800 ms
ERROR: Shader UI/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)

The Linux dedicated Server build (no matter if done with Windows or Linux) throws this:

[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
Preloaded 'libwebrtc.so'
Unable to load player prefs
Segmentation fault (core dumped)

So maybe theres something wrong with linux dedicated server support directly or the compilation of libwebrtcs.so compared to the webrtc.dll on windows?

[karasusan]

@ZenBre4ker Thank you for checking. Maybe libwebrtc.so has something wrong.

[LazyWood]

Excuse me, when will the headless server be supported?

[karasusan]

@LazyWood Sorry but we can't say the date when it will be supported.

[ZenBre4ker]

@karasusan Further investigation shows, that commenting out this line

if (!vulkan->AddInterceptInitialization(InterceptVulkanInitialization, nullptr, 0))
    {
        RTC_LOG(LS_INFO) << "AddInterceptInitialization failed.";
    }

https://github.com/Unity-Technologies/com.unity.webrtc/blob/main/Plugin%7E/WebRTCPlugin/UnityRenderEvent.cpp#L193 lets the program properly initialize before crashing later. Valgrind states, that there is an invalid read of size 8.

The crash later is probably due to the missing vulkanInterception, because the code is hidden inside the UnityPlayer.so, which i dont know how to get Debug Symbols for.

So maybe its rather a problem with vulkan on a dedicated server?

[karasusan]

@ZenBre4ker There is a possibility. Graphics features are stripped when building a dedicated server, so might occurrs a crash on the line which you mentioned.

[cakeslice]

@karasusan Still getting this issue on a dedicated server linux build even with the merged changes from #887:

symbol lookup error: undefined symbol: UnitySetGraphicsDevice (fatal) symbol lookup error: undefined symbol: UnityRenderEvent (fatal) symbol lookup error: undefined symbol: UnityGetAudioEffectDefinitions (fatal) symbol lookup error: undefined symbol: UnityRenderingExtEvent (fatal) symbol lookup error: undefined symbol: UnityRenderingExtQuery (fatal) symbol lookup error: undefined symbol: UnityShaderCompilerExtEvent (fatal)

For my use case, I only want to use the data channel so I don't need audio, video or rendering of any kind.

[ZenBre4ker]

@cakeslice The changes from #887 only change source files, they still need to be compiled into a shared library to work. Also your errors are normal and dont mean that it doesnt work. I fixed only the segmentation fault due to missing vulkan, not any missing symbols.

[cakeslice]

@ZenBre4ker You're right, after compiling it works now. Thank you!