fio fio hangs after ctrl+c when I/Os never return

Please acknowledge the following before creating a ticket

[x] I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug: after ctrl+C to kill fio, fio cann't quit normally, see:

and we use gdb to debug fio, it shows futex_wait_cancelable is waiting for sth.

Environment: Debian

fio version: 3.40

Reproduction steps

Jul 02 '25 06:07 xirs

Hello @xirs: I'm afraid there isn't enough information to start looking at this. For starters could you tell us:

What is the full job file/command line that reproduces this issue?
If the above is more than 5 lines/options can you minimise the job file and command line options (it's important to know them all) such that you have the smallest amount that still reproduce the issue. Don't stop at the first option that is required, put it back and then try to remove the next option and so on.
How often is the problem reproducible? Every time? 50% of time? Never happens again?

(A general plea: please attach text as a markdown formatted code block rather than screenshots. Thanks!)

Jul 02 '25 12:07 sitsofe

I use fio to test a NAS system, and there is weak network fault or remote server fault sometime.

fio hang again today, see fio options: fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=50 -name=write --size=70G --group_reporting

see fio output:

fio version is fio-3.12

Jul 04 '25 06:07 xirs

I use gdb to print all threads stack:

(gdb) thread apply all bt

Thread 50 (Thread 0x7f978ffff700 (LWP 1251746)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 49 (Thread 0x7f97acff9700 (LWP 1251744)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 48 (Thread 0x7f97ad7fa700 (LWP 1251742)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 47 (Thread 0x7f97adffb700 (LWP 1251740)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 46 (Thread 0x7f97ae7fc700 (LWP 1251738)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 45 (Thread 0x7f97aeffd700 (LWP 1251736)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 44 (Thread 0x7f97af7fe700 (LWP 1251734)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 43 (Thread 0x7f97affff700 (LWP 1251732)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 42 (Thread 0x7f97ccff9700 (LWP 1251730)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 41 (Thread 0x7f97cd7fa700 (LWP 1251728)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 40 (Thread 0x7f97cdffb700 (LWP 1251726)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 39 (Thread 0x7f97ce7fc700 (LWP 1251724)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 38 (Thread 0x7f97cf7fe700 (LWP 1251720)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 37 (Thread 0x7f97cffff700 (LWP 1251718)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 36 (Thread 0x7f97ecff9700 (LWP 1251716)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 35 (Thread 0x7f97ed7fa700 (LWP 1251714)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 34 (Thread 0x7f97edffb700 (LWP 1251712)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 33 (Thread 0x7f97ee7fc700 (LWP 1251710)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 32 (Thread 0x7f97eeffd700 (LWP 1251708)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 31 (Thread 0x7f97ef7fe700 (LWP 1251706)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 30 (Thread 0x7f97effff700 (LWP 1251704)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 29 (Thread 0x7f980cff9700 (LWP 1251702)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 28 (Thread 0x7f980d7fa700 (LWP 1251700)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 27 (Thread 0x7f980dffb700 (LWP 1251698)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 26 (Thread 0x7f980e7fc700 (LWP 1251696)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 25 (Thread 0x7f980effd700 (LWP 1251695)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 24 (Thread 0x7f980f7fe700 (LWP 1251694)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7f980ffff700 (LWP 1251693)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7f982cff9700 (LWP 1251692)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7f982d7fa700 (LWP 1251691)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f982dffb700 (LWP 1251690)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f982e7fc700 (LWP 1251689)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f982effd700 (LWP 1251688)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f982f7fe700 (LWP 1251687)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f982ffff700 (LWP 1251686)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f984cff9700 (LWP 1251685)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f984d7fa700 (LWP 1251684)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f984dffb700 (LWP 1251683)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f984e7fc700 (LWP 1251682)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f984effd700 (LWP 1251681)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f984f7fe700 (LWP 1251680)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f984ffff700 (LWP 1251679)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f986cff9700 (LWP 1251678)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f986d7fa700 (LWP 1251677)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f986dffb700 (LWP 1251676)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
--Type <RET> for more, q to quit, c to continue without paging--
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f986effd700 (LWP 1251674)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f986f7fe700 (LWP 1251673)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f986ffff700 (LWP 1251672)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eb5054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eb5000, cond=0x7f9895eb5028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eb5028, mutex=0x7f9895eb5000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab23b3fc in ?? ()
#5  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f9874ad0700 (LWP 1251671)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7f9895eac054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f9895eac000, cond=0x7f9895eac028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f9895eac028, mutex=0x7f9895eac000) at pthread_cond_wait.c:655
#3  0x000055faab21d4b9 in fio_sem_down ()
#4  0x000055faab203b86 in __show_running_run_stats ()
#5  0x000055faab2455cd in ?? ()
#6  0x00007f98a1673fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f98a159e06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f9895f00f00 (LWP 1251669)):
#0  0x00007f98a156b5c0 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffeb23b7160, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f98a1596414 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32
#2  0x000055faab23cca0 in ?? ()
#3  0x000055faab23d2ad in fio_backend ()
#4  0x000055faab1dead1 in main ()
(gdb) 
(gdb)

Jul 04 '25 06:07 xirs

@xirs If the underlying kernel device or filesystem (in your case a network share) hangs then fio will wait for outstanding I/Os to come back forever. Many I/O calls don't have a way to say "abandon these I/Os forever so that waiters return" so at that point a userspace program is permanently waiting.

Put another way: if you ask fio to do a clean shutdown and it can't be done then there's nothing fio can do... You can always send fio a SIGKILL or SIGQUIT and live with the consequences of potentially leftover state.

Jul 04 '25 07:07 sitsofe

So two questions:

Can you make the problem happen with a smaller numjobs (e.g. something less than 8 and the smaller the better)
Are any of your fio processes in a D state after the ctrl-c?

Jul 04 '25 07:07 sitsofe

Additionally can you:

Minimise the job file and command line options such that you have the smallest amount of options AND they have the smallest possible values such that still reproduce the issue. Don't stop at the first option that is required, put it back and then try to remove the next option and so on.
Compile an unoptimised version of fio (./configure --disable-optimizations) and reproduce the problem with the above reduced options as that should make the gdb output more useful.

Jul 04 '25 10:07 sitsofe

(BTW: an iodepth greater than 1 doesn't make sense when you are using a synchronous ioengine like sync)

Jul 07 '25 08:07 sitsofe

after kill fio, all of its thread are ijn SLEEP status, no D state

Jul 07 '25 10:07 xirs

(BTW: an iodepth greater than 1 doesn't make sense when you are using a synchronous ioengine like sync)

but I dont't think iodepth paramter will bring to IO hang since it is useless when using IO engine like sync

Jul 07 '25 10:07 xirs

my question is that if there is in flight IO to bring IO hang, FIO thread stack should be at a waitingt "something to return" status. but I cant' see a thread in this status, all of threads are in sleep. So what's wrong about fio?

Jul 07 '25 10:07 xirs

I will try compile fio-3.40 tags with './configure --disable-optimizations' , and reproduce the issue in my env

Jul 07 '25 10:07 xirs

I compile fio-3.40 with './configure --disable-optimizations'.

fio hang again, see gdb 'thread apply all bt' output

Type "apropos word" to search for commands related to "word".
Attaching to process 2533776
[New LWP 2533810]
[New LWP 2533820]
[New LWP 2533821]
[New LWP 2533822]
[New LWP 2533823]
[New LWP 2533824]
[New LWP 2533826]
[New LWP 2533956]
[New LWP 2533983]
[New LWP 2533984]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fc9456455c0 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffec5aafe10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28      ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
(gdb) **thread apply all bt**

Thread 11 (Thread 0x7fc909494700 (LWP 2533984)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fe0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fc909c95700 (LWP 2533983)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9ff0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fc9174b0700 (LWP 2533956)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fd0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fc93a4ec700 (LWP 2533826)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fe0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fc93b4f0700 (LWP 2533824)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fd0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fc93bcf2700 (LWP 2533823)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fe0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fc92dcdd700 (LWP 2533822)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9ff0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fc92d4dc700 (LWP 2533821)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9ff0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fc92ccdb700 (LWP 2533820)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93bcf8054) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93bcf8000, cond=0x7fc93bcf8028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93bcf8028, mutex=0x7fc93bcf8000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93bcf8000) at fio_sem.c:157
#4  0x0000560b337997de in thread_main (data=0x560b35da9fe0) at backend.c:2022
#5  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

--Type <RET> for more, q to quit, c to continue without paging--
Thread 2 (Thread 0x7fc92c4da700 (LWP 2533810)):
#0  futex_wait_cancelable (private=128, expected=0, futex_word=0x7fc93b4f1050) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fc93b4f1000, cond=0x7fc93b4f1028) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fc93b4f1028, mutex=0x7fc93b4f1000) at pthread_cond_wait.c:655
#3  0x0000560b3376daf6 in fio_sem_down (sem=0x7fc93b4f1000) at fio_sem.c:157
#4  0x0000560b3374d1d3 in __show_running_run_stats () at stat.c:2788
#5  0x0000560b337a7b8b in eval_timer (it=0x7fc92c4d9df8, now=0x7fc92c4d9dc0, msec_to_next_event=0x7fc92c4d9e50) at helper_thread.c:270
#6  0x0000560b337a7e05 in helper_thread_main (data=0x7fc9443fac90) at helper_thread.c:338
#7  0x00007fc9458d5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fc94567806f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fc9453dc7c0 (LWP 2533776)):
#0  0x00007fc9456455c0 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffec5aafe10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007fc945670414 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32
#2  0x0000560b3379a3d6 in do_usleep (usecs=10000) at backend.c:2317
#3  0x0000560b3379b0af in run_threads (sk_out=0x0) at backend.c:2640
#4  0x0000560b3379b2b5 in fio_backend (sk_out=0x0) at backend.c:2696
#5  0x0000560b337cf42a in main (argc=21, argv=0x7ffec5ab80e8, envp=0x7ffec5ab8198) at fio.c:58
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb)

Jul 09 '25 03:07 xirs

and FIO last out is

fio-3.40
Starting 100 threads
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533828, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533825, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533831, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533830, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533834, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533829, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533833, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533832, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533838, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533835, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533836, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533837, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533839, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533840, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533841, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533827, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533842, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533844, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533847, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533843, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533846, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533848, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533849, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533850, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533851, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533845, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533852, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533853, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533854, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533855, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533922, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533923, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533925, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533927, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533924, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533929, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533930, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533928, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533932, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533934, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533933, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533926, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533856, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533931, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533935, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533938, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533937, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533941, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533936, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533940, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533942, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533944, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533943, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533947, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533948, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533949, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533946, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533950, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533951, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533952, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533939, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533954, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533953, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533955, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533945, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533957, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533958, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533959, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533960, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533962, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533963, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533965, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533964, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533961, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533966, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533968, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533967, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533969, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533970, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533971, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533972, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533974, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533973, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533975, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533976, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533977, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533980, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533979, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533981, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533978, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=2533982, err=35/

write5MB: (groupid=0, jobs=100): err=35 (): pid=2533820: Wed Jul  9 05:38:53 2025
  write: IOPS=21, BW=1292KiB/s (1323kB/s)(30.0MiB/23771msec); 0 zone resets
    clat (usec): min=1295, max=10015k, avg=114068.46, stdev=692085.57
     lat (msec): min=5, max=10018, avg=118.28, stdev=692.12
    clat percentiles (usec):
     |  1.00th=[   1319],  5.00th=[   1369], 10.00th=[   1385],
     | 20.00th=[   1434], 30.00th=[   1467], 40.00th=[   1516],
     | 50.00th=[   1582], 60.00th=[   1647], 70.00th=[   1811],
     | 80.00th=[   2212], 90.00th=[  57410], 95.00th=[ 513803],
     | 99.00th=[2332034], 99.50th=[6140462], 99.90th=[9999221],
     | 99.95th=[9999221], 99.99th=[9999221]
   bw (  KiB/s): min=43008, max=49152, per=100.00%, avg=46080.00, stdev=1942.90, samples=6
   iops        : min=   42, max=   48, avg=45.00, stdev= 1.90, samples=6
  lat (usec)   : 500=1.17%, 750=0.19%
  lat (msec)   : 2=75.29%, 4=5.84%, 10=3.11%, 20=0.19%, 50=2.53%
  lat (msec)   : 100=1.17%, 250=2.92%, 500=1.56%, 750=2.33%, 1000=1.36%
  lat (msec)   : 2000=0.78%, >=2000=1.56%
  cpu          : usr=0.04%, sys=0.00%, ctx=765, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=14,500,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=1292KiB/s (1323kB/s), 1292KiB/s-1292KiB/s (1323kB/s-1323kB/s), io=30.0MiB (31.5MB), run=23771-23771msec

Jul 09 '25 03:07 xirs

@sitsofe please help analyse this, thank you in advance

Jul 09 '25 03:07 xirs

another fio also hang, the GDB output is

ype "apropos word" to search for commands related to "word".
Attaching to process 2943158
[New LWP 2943184]
[New LWP 2943224]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f50fc6787f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffe18fbb8b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78	../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) thread apply all bt

Thread 3 (Thread 0x7f50f0e8c640 (LWP 2943224) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f50f3e9a054) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f50f3e9a054) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f50f3e9a054, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f50fc626a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f50f3e9a000, cond=0x7f50f3e9a028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f50f3e9a028, mutex=0x7f50f3e9a000) at ./nptl/pthread_cond_wait.c:627
#5  0x000055a2b162e20e in fio_sem_down (sem=0x7f50f3e9a000) at fio_sem.c:157
#6  0x000055a2b165cf4c in thread_main (data=0x55a2b2149960) at backend.c:2022
#7  0x00007f50fc627ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f50fc6b9850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f50f3e98640 (LWP 2943184) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f50f0e8d050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f50f0e8d050) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f50f0e8d050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f50fc626a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f50f0e8d000, cond=0x7f50f0e8d028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f50f0e8d028, mutex=0x7f50f0e8d000) at ./nptl/pthread_cond_wait.c:627
#5  0x000055a2b162e20e in fio_sem_down (sem=0x7f50f0e8d000) at fio_sem.c:157
#6  0x000055a2b160b5d4 in __show_running_run_stats () at stat.c:2788
#7  0x000055a2b166b4a5 in eval_timer (it=0x7f50f3e97df8, now=0x7f50f3e97dc0, msec_to_next_event=0x7f50f3e97d98) at helper_thread.c:270
#8  0x000055a2b166b773 in helper_thread_main (data=0x7f50fb5a4c50) at helper_thread.c:338
#9  0x00007f50fc627ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f50fc6b9850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f50fc5922c0 (LWP 2943158) "fio"):
#0  0x00007f50fc6787f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffe18fbb8b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007f50fc67d677 in __GI___nanosleep (req=req@entry=0x7ffe18fbb8b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007f50fc6aef2f in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x000055a2b165dc1c in do_usleep (usecs=10000) at backend.c:2317
#4  0x000055a2b165eaee in run_threads (sk_out=0x0) at backend.c:2640
#5  0x000055a2b165ed2e in fio_backend (sk_out=0x0) at backend.c:2696
#6  0x000055a2b15dc121 in main (argc=21, argv=0x7ffe18fc3b88, envp=<optimized out>) at fio.c:58
(gdb) 
quit

Jul 09 '25 03:07 xirs

fio hang agagin, see last output:

see fio threads status


root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     4149589 4149589  0.1  4.9 6825776 6530236 ?     Sl   Jul13   2:49 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149622  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149633  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149635  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149638  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149639  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100
root     4149589 4149728  0.0  4.9 6825776 6530236 ?     Sl   Jul13   0:00 fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 -thread -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=100

and after kill -9 fio, it quit successfully

root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# 
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# kill -9 4149589
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     4149589 4149589  0.1  0.0      0     0 ?        Zl   Jul13   2:49 [fio] <defunct>
root     4149589 4149728  0.0  0.0      0     0 ?        Dl   Jul13   0:00 [fio]
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     4149589 4149589  0.1  0.0      0     0 ?        Zl   Jul13   2:49 [fio] <defunct>
root     4149589 4149728  0.0  0.0      0     0 ?        Dl   Jul13   0:00 [fio]
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     4149589 4149589  0.1  0.0      0     0 ?        Zl   Jul13   2:49 [fio] <defunct>
root     4149589 4149728  0.0  0.0      0     0 ?        Dl   Jul13   0:00 [fio]
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     4149589 4149589  0.1  0.0      0     0 ?        Zl   Jul13   2:49 [fio] <defunct>
root     4149589 4149728  0.0  0.0      0     0 ?        Dl   Jul13   0:00 [fio]
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root@ECS-shenjunwe-fsx-for-benz:~/fsx_long_test# ps u  -T -p 4149589
USER         PID    SPID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

Jul 14 '25 02:07 xirs

@xirs :

OK the latest backtrace looks like it happened with a single job - is this correct? If so can you post the cut down job? Does this happen when you use a local non-network filesystem too?

I've tried reproducing this locally but I've had no luck. Just for reference what kernel is being used? It may also be interesting to see the --debug=mutex output of a the smaller run that gets stuck.

Jul 14 '25 09:07 sitsofe

I'll try run fio with --debug=mutex, please wait.

"the latest backtrace looks like it happened with a single job - is this correct? If so can you post the cut down job?" I didn't quite fully understand what your meanning, I'm not sure what I should do
Does this happen when you use a local non-network filesystem too? I couldn't reproduce this too

Jul 23 '25 07:07 xirs

@xirs: Re latest backtrace looks like it happened with a single job: The backtrace you gave in https://github.com/axboe/fio/issues/1934#issuecomment-3050981640 is short (only three threads) and which makes me wonder if it produced when you were using fewer jobs. Is this the case and if so just how many jobs were involved?

I guess while I'm here: which version of Debian are you using, are you running on the x86_64 architecture and can you still reproduce the problem when you use --thread?

Jul 23 '25 14:07 sitsofe

see my latest run with --debug=mutex and --thread

fio --eta-newline=1 --status-interval=60 -directory=/mnt/fuse_disk/disktest -direct=1 -iodepth=64 **-thread** -rw=write --verify=crc64 --file_append=0 --create_on_open=1 --allrandrepeat=1 --invalidate=1 --fadvise_hint=0 -ioengine=sync -bs=1MB --allrandrepeat=1 -numjobs=6   -name=write5MB  --size=5MB --group_reporting

 

fio: set debug option mutex
write5MB: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=64
...
fio-3.40
mutex    1321576 wait on startup_sem
mutex    1321576 done waiting on startup_sem
Starting 100 threads
mutex    1321576 wait on startup_sem
mutex    1321722 up startup_sem
mutex    1321722 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321723 up startup_sem
mutex    1321723 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321724 up startup_sem
mutex    1321724 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321725 up startup_sem
mutex    1321725 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321726 up startup_sem
mutex    1321726 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321727 up startup_sem
mutex    1321727 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321728 up startup_sem
mutex    1321728 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321729 up startup_sem
mutex    1321729 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321730 up startup_sem
mutex    1321730 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321731 up startup_sem
mutex    1321731 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321732 up startup_sem
mutex    1321732 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321733 up startup_sem
mutex    1321733 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321734 up startup_sem
mutex    1321734 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321735 up startup_sem
mutex    1321735 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321736 up startup_sem
mutex    1321736 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321737 up startup_sem
mutex    1321737 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321738 up startup_sem
mutex    1321738 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321739 up startup_sem
mutex    1321739 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321740 up startup_sem
mutex    1321740 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321741 up startup_sem
mutex    1321741 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321742 up startup_sem
mutex    1321742 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321743 up startup_sem
mutex    1321743 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321744 up startup_sem
mutex    1321744 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321745 up startup_sem
mutex    1321745 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321746 up startup_sem
mutex    1321746 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321747 up startup_sem
mutex    1321747 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321748 up startup_sem
mutex    1321748 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321749 up startup_sem
mutex    1321749 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321750 up startup_sem
mutex    1321750 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321751 up startup_sem
mutex    1321751 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321752 up startup_sem
mutex    1321752 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321753 up startup_sem
mutex    1321753 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321754 up startup_sem
mutex    1321754 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321755 up startup_sem
mutex    1321755 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321756 up startup_sem
mutex    1321756 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321757 up startup_sem
mutex    1321757 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321758 up startup_sem
mutex    1321758 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321759 up startup_sem
mutex    1321759 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321760 up startup_sem
mutex    1321760 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321761 up startup_sem
mutex    1321761 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321762 up startup_sem
mutex    1321762 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321763 up startup_sem
mutex    1321763 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321764 up startup_sem
mutex    1321764 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321765 up startup_sem
mutex    1321765 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321766 up startup_sem
mutex    1321766 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321767 up startup_sem
mutex    1321767 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321768 up startup_sem
mutex    1321768 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321769 up startup_sem
mutex    1321769 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321770 up startup_sem
mutex    1321770 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321771 up startup_sem
mutex    1321771 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321772 up startup_sem
mutex    1321772 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321773 up startup_sem
mutex    1321773 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321774 up startup_sem
mutex    1321774 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321775 up startup_sem
mutex    1321775 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321776 up startup_sem
mutex    1321776 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321777 up startup_sem
mutex    1321777 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321778 up startup_sem
mutex    1321778 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321779 up startup_sem
mutex    1321779 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321780 up startup_sem
mutex    1321780 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321781 up startup_sem
mutex    1321781 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321782 up startup_sem
mutex    1321782 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321783 up startup_sem
mutex    1321783 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321784 up startup_sem
mutex    1321784 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321785 up startup_sem
mutex    1321785 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321786 up startup_sem
mutex    1321786 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321787 up startup_sem
mutex    1321787 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321788 up startup_sem
mutex    1321788 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321789 up startup_sem
mutex    1321789 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321790 up startup_sem
mutex    1321790 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321791 up startup_sem
mutex    1321791 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321792 up startup_sem
mutex    1321792 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321793 up startup_sem
mutex    1321793 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321794 up startup_sem
mutex    1321794 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321795 up startup_sem
mutex    1321795 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321796 up startup_sem
mutex    1321796 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321797 up startup_sem
mutex    1321797 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321798 up startup_sem
mutex    1321798 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321799 up startup_sem
mutex    1321799 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321800 up startup_sem
mutex    1321800 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321801 up startup_sem
mutex    1321801 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321802 up startup_sem
mutex    1321802 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321803 up startup_sem
mutex    1321803 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321804 up startup_sem
mutex    1321804 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321805 up startup_sem
mutex    1321805 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321806 up startup_sem
mutex    1321806 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321807 up startup_sem
mutex    1321807 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321808 up startup_sem
mutex    1321808 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321809 up startup_sem
mutex    1321809 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321810 up startup_sem
mutex    1321810 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321811 up startup_sem
mutex    1321811 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321812 up startup_sem
mutex    1321812 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321813 up startup_sem
mutex    1321813 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321814 up startup_sem
mutex    1321814 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321815 up startup_sem
mutex    1321815 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321816 up startup_sem
mutex    1321816 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321817 up startup_sem
mutex    1321817 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321818 up startup_sem
mutex    1321818 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321819 up startup_sem
mutex    1321819 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321820 up startup_sem
mutex    1321820 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321576 wait on startup_sem
mutex    1321821 up startup_sem
mutex    1321821 wait on td->sem
mutex    1321576 done waiting on startup_sem
mutex    1321724 done waiting on td->sem
mutex    1321731 done waiting on td->sem
mutex    1321735 done waiting on td->sem
mutex    1321725 done waiting on td->sem
mutex    1321746 done waiting on td->sem
mutex    1321749 done waiting on td->sem
mutex    1321750 done waiting on td->sem
mutex    1321753 done waiting on td->sem
mutex    1321756 done waiting on td->sem
mutex    1321759 done waiting on td->sem
mutex    1321760 done waiting on td->sem
mutex    1321762 done waiting on td->sem
mutex    1321766 done waiting on td->sem
mutex    1321768 done waiting on td->sem
mutex    1321773 done waiting on td->sem
mutex    1321771 done waiting on td->sem
mutex    1321778 done waiting on td->sem
mutex    1321782 done waiting on td->sem
mutex    1321786 done waiting on td->sem
mutex    1321791 done waiting on td->sem
mutex    1321790 done waiting on td->sem
mutex    1321741 done waiting on td->sem
mutex    1321795 done waiting on td->sem
mutex    1321796 done waiting on td->sem
mutex    1321798 done waiting on td->sem
mutex    1321747 done waiting on td->sem
mutex    1321748 done waiting on td->sem
mutex    1321806 done waiting on td->sem
mutex    1321740 done waiting on td->sem
mutex    1321752 done waiting on td->sem
mutex    1321817 done waiting on td->sem
mutex    1321815 done waiting on td->sem
mutex    1321819 done waiting on td->sem
mutex    1321820 done waiting on td->sem
mutex    1321813 done waiting on td->sem
mutex    1321723 done waiting on td->sem
mutex    1321757 done waiting on td->sem
mutex    1321733 done waiting on td->sem
mutex    1321758 done waiting on td->sem
mutex    1321730 done waiting on td->sem
mutex    1321761 done waiting on td->sem
mutex    1321732 done waiting on td->sem
mutex    1321764 done waiting on td->sem
mutex    1321765 done waiting on td->sem
mutex    1321736 done waiting on td->sem
mutex    1321767 done waiting on td->sem
mutex    1321763 done waiting on td->sem
mutex    1321769 done waiting on td->sem
mutex    1321770 done waiting on td->sem
mutex    1321734 done waiting on td->sem
mutex    1321772 done waiting on td->sem
mutex    1321774 done waiting on td->sem
mutex    1321775 done waiting on td->sem
mutex    1321722 done waiting on td->sem
mutex    1321776 done waiting on td->sem
mutex    1321737 done waiting on td->sem
mutex    1321779 done waiting on td->sem
mutex    1321780 done waiting on td->sem
mutex    1321781 done waiting on td->sem
mutex    1321738 done waiting on td->sem
mutex    1321783 done waiting on td->sem
mutex    1321785 done waiting on td->sem
mutex    1321784 done waiting on td->sem
mutex    1321739 done waiting on td->sem
mutex    1321787 done waiting on td->sem
mutex    1321789 done waiting on td->sem
mutex    1321788 done waiting on td->sem
mutex    1321792 done waiting on td->sem
mutex    1321742 done waiting on td->sem
mutex    1321794 done waiting on td->sem
mutex    1321743 done waiting on td->sem
mutex    1321793 done waiting on td->sem
mutex    1321797 done waiting on td->sem
mutex    1321744 done waiting on td->sem
mutex    1321799 done waiting on td->sem
mutex    1321745 done waiting on td->sem
mutex    1321800 done waiting on td->sem
mutex    1321801 done waiting on td->sem
mutex    1321802 done waiting on td->sem
mutex    1321803 done waiting on td->sem
mutex    1321804 done waiting on td->sem
mutex    1321805 done waiting on td->sem
mutex    1321807 done waiting on td->sem
mutex    1321726 done waiting on td->sem
mutex    1321808 done waiting on td->sem
mutex    1321809 done waiting on td->sem
mutex    1321751 done waiting on td->sem
mutex    1321810 done waiting on td->sem
mutex    1321811 done waiting on td->sem
mutex    1321812 done waiting on td->sem
mutex    1321814 done waiting on td->sem
mutex    1321727 done waiting on td->sem
mutex    1321816 done waiting on td->sem
mutex    1321728 done waiting on td->sem
mutex    1321818 done waiting on td->sem
mutex    1321754 done waiting on td->sem
mutex    1321821 done waiting on td->sem
mutex    1321755 done waiting on td->sem
mutex    1321729 done waiting on td->sem
mutex    1321777 done waiting on td->sem
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321750, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321751, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321813, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321741, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321791, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321752, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321724, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321766, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321748, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321801, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321783, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321756, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321746, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321806, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321754, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321793, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321768, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321807, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321772, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321775, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321788, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321765, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321812, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321743, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321737, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321800, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321802, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321816, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321779, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321769, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321738, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321733, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321771, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321810, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321758, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321809, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321781, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321739, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321745, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321764, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321728, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321722, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321789, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321774, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321805, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321757, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321811, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321804, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321820, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321755, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321763, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321821, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321799, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321736, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321803, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321814, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321818, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321776, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321780, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321727, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321797, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321742, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321767, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321730, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321794, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321787, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321729, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321726, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321770, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321808, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321792, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321723, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321761, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321785, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321744, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321784, err=35/
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321777, err=35/

write5MB: (groupid=0, jobs=100): err=35 (): pid=1321722: Thu Jul 24 03:48:27 2025
  write: IOPS=17, BW=4263KiB/s (4365kB/s)(120MiB/28825msec); 0 zone resets
    clat (usec): min=1331, max=27066k, avg=525916.00, stdev=2697176.60
     lat (msec): min=4, max=27069, avg=529.05, stdev=2697.18
    clat percentiles (usec):
     |  1.00th=[    1385],  5.00th=[    1418], 10.00th=[    1450],
     | 20.00th=[    1516], 30.00th=[    1598], 40.00th=[    1729],
     | 50.00th=[    1893], 60.00th=[    2704], 70.00th=[    4293],
     | 80.00th=[    5473], 90.00th=[    8291], 95.00th=[ 2139096],
     | 99.00th=[16039019], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=188416, max=210944, per=100.00%, avg=198997.33, stdev=1119.90, samples=35
   iops        : min=  184, max=  206, avg=194.33, stdev= 1.09, samples=35
  lat (usec)   : 500=1.82%, 750=0.17%
  lat (msec)   : 2=43.73%, 4=12.38%, 10=18.65%, 20=1.32%, 50=2.48%
  lat (msec)   : 100=1.82%, 250=6.77%, 500=4.29%, 750=1.16%, 1000=0.17%
  lat (msec)   : 2000=0.33%, >=2000=4.95%
  cpu          : usr=0.02%, sys=0.00%, ctx=720, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=110,496,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=4263KiB/s (4365kB/s), 4263KiB/s-4263KiB/s (4365kB/s-4365kB/s), io=120MiB (126MB), run=28825-28825msec
fio seems to be stuck grabbing stat_sem, forcibly exiting
fio: pid=1321734, err=35/

and see gdb output

root@ECS-fsx-for-benz-1:~/fsx_long_test# gdb --pid 1321576
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 1321576
[New LWP 1321674]
[New LWP 1321725]
[New LWP 1321732]
[New LWP 1321735]
[New LWP 1321740]
[New LWP 1321747]
[New LWP 1321749]
[New LWP 1321753]
[New LWP 1321762]
[New LWP 1321773]
[New LWP 1321778]
[New LWP 1321786]
[New LWP 1321790]
[New LWP 1321795]
[New LWP 1321796]
[New LWP 1321798]
[New LWP 1321815]
[New LWP 1321817]
[New LWP 1321819]
BFD: error: /usr/lib/debug/.build-id/cc/afe3dc4ac3456c1d7b9cbeeb21b4dea429eb4b.debug(.debug_info) is too large (0x15f8ad bytes)

warning: Can't read data for section '.debug_info' in file '/usr/lib/debug/.build-id/cc/afe3dc4ac3456c1d7b9cbeeb21b4dea429eb4b.debug'

warning: Section .debug_aranges in /usr/lib/debug/.build-id/cc/afe3dc4ac3456c1d7b9cbeeb21b4dea429eb4b.debug entry at offset 0 debug_info_offset 0 does not exists, ignoring .debug_aranges.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f960c9b3505 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffff1ac1030, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
48      ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) thread apply all bt

Thread 20 (Thread 0x7f95d175a6c0 (LWP 1321819) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc054) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc054, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc054, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 19 (Thread 0x7f95d275c6c0 (LWP 1321817) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 18 (Thread 0x7f95d375e6c0 (LWP 1321815) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 17 (Thread 0x7f95dbf6f6c0 (LWP 1321798) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 16 (Thread 0x7f95dcf716c0 (LWP 1321796) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
--Type <RET> for more, q to quit, c to continue without paging--
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 15 (Thread 0x7f95dd7726c0 (LWP 1321795) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 14 (Thread 0x7f95dff776c0 (LWP 1321790) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x7f95e1f7b6c0 (LWP 1321786) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 12 (Thread 0x7f95e5f836c0 (LWP 1321778) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-int--Type <RET> for more, q to quit, c to continue without paging--
ernal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x7f95e87886c0 (LWP 1321773) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7f95edf936c0 (LWP 1321762) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7f95f279c6c0 (LWP 1321753) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7f95f6f866c0 (LWP 1321749) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f95f7f8a6c0 (LWP 1321747) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f95fb7986c0 (LWP 1321740) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f95fdfa26c0 (LWP 1321735) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f95ff7a86c0 (LWP 1321732) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
--Type <RET> for more, q to quit, c to continue without paging--
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f9602fb66c0 (LWP 1321725) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fbc050) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fbc050, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fbc000, cond=0x7f9602fbc028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fbc028, mutex=mutex@entry=0x7f9602fbc000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fbc000) at fio_sem.c:157
#6  0x0000558f0b1abd4c in thread_main (data=<optimized out>) at backend.c:2022
#7  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f95f379e6c0 (LWP 1321674) "fio"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f9602fb7054) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9602fb7054, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f960c969f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9602fb7054, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f960c96c5d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f9602fb7000, cond=0x7f9602fb7028) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7f9602fb7028, mutex=mutex@entry=0x7f9602fb7000) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558f0b188469 in fio_sem_down (sem=0x7f9602fb7000) at fio_sem.c:157
#6  0x0000558f0b16dee6 in __show_running_run_stats () at stat.c:2788
#7  0x0000558f0b1b9837 in eval_timer (msec_to_next_event=<synthetic pointer>, now=0x7f95f379dc40, it=0x7f95f379dca8) at helper_thread.c:270
#8  helper_thread_main (data=0x7f960b6c3c90) at helper_thread.c:338
#9  0x00007f960c96d1c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f960c9ed85c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f960c6a57c0 (LWP 1321576) "fio"):
#0  0x00007f960c9b3505 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffff1ac1030, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1  0x00007f960c9b7e13 in __GI___nanosleep (req=req@entry=0x7ffff1ac1030, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007f960c9e3205 in usleep (useconds=useconds@entry=10000) at ../sysdeps/posix/usleep.c:31
#3  0x0000558f0b1ad66d in do_usleep (usecs=10000) at backend.c:2317
#4  run_threads (sk_out=sk_out@entry=0x0) at backend.c:2640
#5  0x0000558f0b1adad5 in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2696
#6  0x0000558f0b14a861 in main (argc=22, argv=0x7ffff1ac92a8, envp=<optimized out>) at fio.c:58
(gdb) 
(gdb)

Jul 24 '25 07:07 xirs