Random segment fault after running for a while

[wenxuanjun]

I started wayvnc with wayvnc -g -r -f 120 and then connected to it with RealVNC.

After a client connection lasts for a while (about a dozen hours), a segmentation fault occurs.

uname -a:

6.14.3-1-cachyos-bore #1 SMP PREEMPT_DYNAMIC Tue, 22 Apr 2025 13:37:42 +0000 x86_64 GNU/Linux

wayvnc -V:

wayvnc: v0.10-dev-5b26ba2 (master)
neatvnc: v0.10-dev-a4b2382 (master)
aml: v1.0.0-rc0-1-gf6086bc (master)

Here is the backtrace from coredumpctl gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
b#0  0x0000762b8ef188a4 in try_read_client_message (client=<optimized out>) at ../neatvnc/src/server.c:1940
1940		if (client->net_stream->state == STREAM_STATE_CLOSED)
[Current thread is 1 (Thread 0x762b7c15e140 (LWP 994347))]
(gdb) backtrace
#0  0x0000762b8ef188a4 in try_read_client_message (client=<optimized out>) at ../neatvnc/src/server.c:1940
#1  on_client_event (stream=<optimized out>, event=<optimized out>) at ../neatvnc/src/server.c:2016
#2  0x0000762b8ef16c80 in stream_tcp__on_readable (self=0x5e3c806d3080) at ../neatvnc/src/stream/tcp.c:173
#3  stream_tcp__on_event (handler=<optimized out>) at ../neatvnc/src/stream/tcp.c:204
#4  0x0000762b8ef6f59d in aml__handle_event (self=<optimized out>, obj=0x762b288541e0) at ../aml-git/src/aml.c:928
#5  aml_dispatch (self=0x5e3c80519500) at ../aml-git/src/aml.c:980
#6  0x00005e3c57fa9c19 in main (argc=5, argv=0x7ffcad393b48) at ../src/main.c:2296

[any1]

This looks like it might be the result of unhandled oom. Have you checked if wayvnc leaks resources?

It might not be enough to just run in valgrind or asan. You might need to track memory allocations using a heap analyser like Massif or heaptrack.

[wenxuanjun]

I tried again using heaptrack build/wayvnc -g -r -f 120 and heaptrack stopped tracking automatically after the segfault. But I'm not familiar how to analyze the result.

Here's the original heaptrack dump file.

Edit: The result of gdb backtrace is exactly the same as before.

Edit2: I have 16G RAM and 8G swap. And when wayvnc is killed, there's 6G RAM and 7G swap left.

[any1]

According to the heaptrack file, wayvnc does not leak memory. Come to think of it, oomkiller probably would have kicked in if that were the case.

The quickest way to figure out if memory is being leaked is to look at the "Consumed" graph. If there is an upwards trend, there is a memory leak. But that's not the case.

Can you try building with address sanitiser? This should give use more information about why it's segfaulting. It's done like this: meson setup build -Db_sanitize=address

[wenxuanjun]

Here's the sanitizer's output:

==1779539==ERROR: AddressSanitizer: heap-use-after-free on address 0x6ed86738d2f0 at pc 0x70b87d4b112e bp 0x7fff39a52ff0 sp 0x7fff39a52fe0
READ of size 8 at 0x6ed86738d2f0 thread T0
    #0 0x70b87d4b112d in on_client_event ../neatvnc/src/server.c:2023
    #1 0x70b87d4a459e in stream_tcp__on_readable ../neatvnc/src/stream/tcp.c:173
    #2 0x70b87d4a459e in stream_tcp__on_event ../neatvnc/src/stream/tcp.c:204
    #3 0x70b87dd477de in aml__handle_event ../aml-git/src/aml.c:928
    #4 0x70b87dd477de in aml_dispatch ../aml-git/src/aml.c:980
    #5 0x5a7d9ce4cd97 in main ../src/main.c:2296
    #6 0x70b87d2526b4  (/usr/lib/libc.so.6+0x276b4) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #7 0x70b87d252768 in __libc_start_main (/usr/lib/libc.so.6+0x27768) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #8 0x5a7d9ce3a0c4 in _start (/Build/wayvnc/build/wayvnc+0x1f0c4) (BuildId: 415893b5a43e6412dbc90c37009fec28768e4be8)

0x6ed86738d2f0 is located 496 bytes inside of 4904-byte region [0x6ed86738d100,0x6ed86738e428)
freed by thread T0 here:
    #0 0x70b87d71f8fd in free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:51
    #1 0x70b87d4a5f99 in client_close ../neatvnc/src/server.c:210
    #2 0x70b87d4a718d in security_handshake_failed ../neatvnc/src/auth/common.c:50
    #3 0x70b87d4ad5d0 in on_security_message ../neatvnc/src/server.c:364
    #4 0x70b87d4ad5d0 in try_read_client_message ../neatvnc/src/server.c:1947
    #5 0x70b87d4ad5d0 in on_client_event ../neatvnc/src/server.c:2016
    #6 0x70b87d4a459e in stream_tcp__on_readable ../neatvnc/src/stream/tcp.c:173
    #7 0x70b87d4a459e in stream_tcp__on_event ../neatvnc/src/stream/tcp.c:204
    #8 0x70b87dd477de in aml__handle_event ../aml-git/src/aml.c:928
    #9 0x70b87dd477de in aml_dispatch ../aml-git/src/aml.c:980
    #10 0x5a7d9ce4cd97 in main ../src/main.c:2296
    #11 0x70b87d2526b4  (/usr/lib/libc.so.6+0x276b4) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #12 0x70b87d252768 in __libc_start_main (/usr/lib/libc.so.6+0x27768) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #13 0x5a7d9ce3a0c4 in _start (/Build/wayvnc/build/wayvnc+0x1f0c4) (BuildId: 415893b5a43e6412dbc90c37009fec28768e4be8)

previously allocated by thread T0 here:
    #0 0x70b87d72073d in calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:74
    #1 0x70b87d4be010 in on_connection ../neatvnc/src/server.c:2040
    #2 0x70b87dd477de in aml__handle_event ../aml-git/src/aml.c:928
    #3 0x70b87dd477de in aml_dispatch ../aml-git/src/aml.c:980
    #4 0x5a7d9ce4cd97 in main ../src/main.c:2296
    #5 0x70b87d2526b4  (/usr/lib/libc.so.6+0x276b4) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #6 0x70b87d252768 in __libc_start_main (/usr/lib/libc.so.6+0x27768) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35)
    #7 0x5a7d9ce3a0c4 in _start (/Build/wayvnc/build/wayvnc+0x1f0c4) (BuildId: 415893b5a43e6412dbc90c37009fec28768e4be8)

SUMMARY: AddressSanitizer: heap-use-after-free ../neatvnc/src/server.c:2023 in on_client_event
Shadow bytes around the buggy address:
  0x6ed86738d000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x6ed86738d080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x6ed86738d100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x6ed86738d280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd
  0x6ed86738d300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x6ed86738d500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1779539==ABORTING

[layercak3]

Oops, we missed on_security_message when fixing the UAFs after client close.

If you return -1 instead of sizeof(type) after the security_handshake_failed calls it should stop crashing. But if you reached that function it means the client disconnected and reconnected, and the log would mention that.