open(), rename(), fstat() -> ENOENT (v2)
From #8443, because a validated reproduction case with a described environment should be sufficient.
Ok, finally got around to look at this myself on a windows machine. Can confirm that this (whatever the root cause is) is still happening on
Edition Windows 11 Enterprise
Version 23H2
Installed on 10.11.2022
OS build 22631.3672
Experience Windows Feature Experience Pack 1000.22700.1009.0
Last Update: KB5037853
I can confirm the observation from @holzerseb that it only happens if you mount a directory from the host file system into the
container, a volume is not sufficient:
$ docker volume create storage $ docker run --rm -v "storage:/tmp" quay.io/danielgblack/wsl8443:ubunu2204 ./wsl8443 file /tmp/testfile opened - fd 3 stat fd 3 successfulThis is likely because volumes are hosted on the "normal" filesystem of the WSL distro hosting docker engine (c.f. https://stackoverflow.com/a/64430683/4087068), which would use a "normal" block device filesystem driver like
ext4.If using rancher desktop, which also uses moby (the docker container runtime), it's
\\wsl$\rancher-desktop-data\var\lib\docker\volumes\storage\_data. I checked the mountpoint for that and as expected:$ docker run --rm -v "volume:/tmp" docker.io/bash sh -c "mount | grep /tmp" /dev/sdf on /tmp type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)If you compare that to a mounted path, you'll see that it's using the
9pfilesystem driver instead:$ mkdir hostdir $ docker run --rm -v "$($(pwd).Path)\hostdir:/tmp" docker.io/bash sh -c "mount | grep /tmp" C:\ on /tmp type 9p (rw,dirsync,noatime,aname=drvfs;path=C:\;uid=0;gid=0;metadata;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=4,wfd=4)And running the reproducer above indeed results in the expected "error on fstat errno 2". The full strace I get there is:
$ mkdir hostdir $ docker run --rm -v "$($(pwd).Path)\hostdir:/tmp" quay.io/danielgblack/wsl8443:ubunu2204 ./wsl8443 file /tmp/testfile opened - fd 3 error on fstat errno 2 $ docker run --rm -v "$($(pwd).Path)\hostdir:/tmp" quay.io/danielgblack/wsl8443:ubunu2204 strace ./wsl8443 execve("./wsl8443", ["./wsl8443"], 0x7ffe38267590 /* 3 vars */) = 0 brk(NULL) = 0x55b6b28d7000 arch_prctl(0x3001 /* ARCH_??? */, 0x7fff8d629830) = -1 EINVAL (Invalid argument) mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0f1586a000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=8735, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 8735, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0f15867000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\237\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\244;\374\204(\337f#\315I\214\234\f\256\271\32"..., 68, 896) = 68 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2216304, ...}, AT_EMPTY_PATH) = 0 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 2260560, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0f1563f000 mmap(0x7f0f15667000, 1658880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f0f15667000 mmap(0x7f0f157fc000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7f0f157fc000 mmap(0x7f0f15854000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x214000) = 0x7f0f15854000 mmap(0x7f0f1585a000, 52816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0f1585a000 close(3) = 0 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0f1563c000 arch_prctl(ARCH_SET_FS, 0x7f0f1563c740) = 0 set_tid_address(0x7f0f1563ca10) = 9 set_robust_list(0x7f0f1563ca20, 24) = 0 rseq(0x7f0f1563d0e0, 0x20, 0, 0x53053053) = 0 mprotect(0x7f0f15854000, 16384, PROT_READ) = 0 mprotect(0x55b6b1a7f000, 4096, PROT_READ) = 0 mprotect(0x7f0f158a4000, 8192, PROT_READ) = 0 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 munmap(0x7f0f15867000, 8735) = 0 openat(AT_FDCWD, "/tmp/testfile", O_RDWR|O_CREAT|O_EXCL, 0660) = 3 write(2, "file /tmp/testfile opened - fd 3"..., 33file /tmp/testfile opened - fd 3 ) = 33 rename("/tmp/testfile", "/tmp/movedtestfile") = 0 newfstatat(3, "", 0x7fff8d629840, AT_EMPTY_PATH) = -1 ENOENT (No such file or directory) write(2, "error on fstat errno 2\n", 23error on fstat errno 2 ) = 23 exit_group(1) = ? +++ exited with 1 +++So given the reproducer works, there are two options:
MariaDB relies on unspecified behavior and this reproducer just does the same wrong thing
The networked filesystem 9P has a bug, either in the driver in the kernel or on the windows side of WSL.
Hopefully this helps @grooverdan and/or the WSL team to narrow down the issue.
Edit:
$ wsl -v WSL-Version: 2.1.5.0 Kernelversion: 5.15.146.1-2 WSLg-Version: 1.0.60 MSRDC-Version: 1.2.5105 Direct3D-Version: 1.611.1-81528511 DXCore-Version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows-Version: 10.0.22631.3672
Originally posted by @Xiphoseer in #8443
Logs are required for review from WSL team
If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'. Otherwise please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.
How to collect WSL logs
Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:
Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1
The script will output the path of the log file once done.
If this is a networking issue, please use collect-networking-logs.ps1, following the instructions here
Once completed please upload the output files to this Github issue.
Click here for more info on logging If you choose to email these logs instead of attaching to the bug, please send them to [email protected] with the number of the github issue in the subject, and in the message a link to your comment in the github issue and reply with '/emailed-logs'.
This issue has been automatically closed since it has not had any author activity for the past 7 days. If you're still experiencing this issue please re-file it as a new issue.
Thank you!