sftp-server.exe Sleeps indefinitely on multiple opened files

[fgarant]

trafficstars

Using the tip of the repo

stp-server.exe freezes on an infinite sleep when trying to open multiple files. Here is the stack:

00 000000ff`67af53c8 00007ff9`624b9c21 ntdll!ZwDelayExecution+0x14
*** WARNING: Unable to verify checksum for C:\Program Files\OpenSSH\sftp-server.exe
01 000000ff`67af53d0 00007ff7`bac0a058 KERNELBASE!SleepEx+0xa1
02 000000ff`67af5470 00007ff7`bac06378 sftp_server!wait_for_multiple_objects_enhanced+0xc8 [d:\dev\git\openssh-portable\contrib\win32\win32compat\signal_wait.c @ 97] 
03 000000ff`67af57a0 00007ff7`bac00dda sftp_server!wait_for_any_event+0x228 [d:\dev\git\openssh-portable\contrib\win32\win32compat\signal.c @ 289] 
04 000000ff`67af7840 00007ff7`babf9281 sftp_server!w32_select+0xcaa [d:\dev\git\openssh-portable\contrib\win32\win32compat\w32fd.c @ 843] 
05 000000ff`67afba70 00007ff7`babf1cc7 sftp_server!sftp_server_main+0xab1 [d:\dev\git\openssh-portable\sftp-server.c @ 1879] 
06 000000ff`67affc20 00007ff7`babf9c34 sftp_server!main+0x87 [d:\dev\git\openssh-portable\sftp-server-main.c @ 54] 
07 000000ff`67affc60 00007ff7`bac1b424 sftp_server!wmain+0x174 [d:\dev\git\openssh-portable\contrib\win32\win32compat\wmain_common.c @ 62] 
08 000000ff`67affcd0 00007ff7`bac1b337 sftp_server!invoke_main+0x34 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 80] 
09 000000ff`67affd10 00007ff7`bac1b1fe sftp_server!__scrt_common_main_seh+0x127 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253] 
0a 000000ff`67affd70 00007ff7`bac1b439 sftp_server!__scrt_common_main+0xe [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 296] 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\windows\System32\KERNEL32.DLL - 
0b 000000ff`67affda0 00007ff9`63117974 sftp_server!wmainCRTStartup+0x9 [f:\dd\vctools\crt\vcstartup\src\startup\exe_wmain.cpp @ 17] 
0c 000000ff`67affdd0 00007ff9`65aaa271 KERNEL32!BaseThreadInitThunk+0x14
0d 000000ff`67affe00 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Here is the context of the wait_for_multiple_object_enhanced, we can see that it waits forever with an infinite TIMEOUT:

0:003> dx Debugger.Sessions[0].Processes[23636].Threads[7920].Stack.Frames[2].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[23636].Threads[7920].Stack.Frames[2].SwitchTo()
000000ff`67af5758 unsigned long wait_ret = 0xcccccccc
000000ff`67af57a0 unsigned long nCount = 0
000000ff`67af57a8 void ** lpHandles = 0x000000ff`67af57f0
000000ff`67af57b0 unsigned long dwMilliseconds = 0xffffffff
000000ff`67af57b8 int bAlertable = 0n1
000000ff`67af54a8 unsigned long return_value = 0xffffffff
000000ff`67af54a0 unsigned long bin_size = 0x40
000000ff`67af54a4 unsigned long bins_total = 0x4000000
000000ff`67af54b0 void * wait_event = 0x00000000`00000000
000000ff`67af54d0 struct _wait_for_multiple_objects_struct [16] wait_bins = struct _wait_for_multiple_objects_struct [16]
000000ff`67af5754 unsigned long wait_ret = 0xcccccccc

Use the following bash script to open multiple files simultaneously. (Open 2000+).

export LD_PRELOAD=

walk_dir () {
    shopt -s nullglob dotglob


    for pathname in "$1"/*; do
        if [ -d "$pathname" ]; then
            walk_dir "$pathname"
        else
            #echo "$pathname"
            cat "$pathname" > /dev/null &
            (( wFileCounter++ ))
            if [ `expr $wFileCounter % 10` -eq 0 ]; then
              echo -ne "\rOpened files: $wFileCounter"
            fi
        fi
    done
}

wFileCounter=0

wDirectory1= #Enter a directory here with multiple folders and files

walk_dir "$wDirectory1"

RED='\033[0;31m'
NC='\033[0m'

echo -e "\rTotal files ${RED} $wFileCounter ${NC}"

[erbeone]

When running SSHD on Windows Server 2016 and mounting with sshfs a windows folder on a linux, we have also seen this type of hang when accessing files from that said mounted folder when starting multiple applications.

[NoMoreFood]

I'll try to take a look at this in the next few weeks.

[NoMoreFood]

Do you know roughly the minimum number of simultaneous connections is necessary to reproduce this? It appears that MAXIMUM_WAIT_OBJECTS_ENHANCED is set to 1024 in the current code. It's possible there is a bug in wait_for_multiple_objects_enhanced() or its use within OpenSSH, but wondering if we're just exceeding that limit.