Dir.statFile now uses fstatat (fewer syscalls)

[KaneRoot]

trafficstars

Hello.

This PR:

• changes Dir.statFile to use a single syscall (fstatat, called in the os.fstatat function)
• adds functions in File.Stat to initiate it from an os.system.Stat structure (given by os.fstatat)
• mainly keeps the Dir.statFile signature, except for errors that are now inherited from os.fstatat
• adds Dir.fstatat in order to avoid the conversion from os.system.Stat to File.Stat, which drops data such as uid and gid

Also, there is currently no test for this function in the standard library. Tell me how to test Dir.statFile and I'll add it in the PR, if you want.

Thanks for your time!

[KaneRoot]

Just commenting on consistency with the rest of the file, sorry if it's too much noise at this stage 😬

No that's okay. I just pushed your suggestions.

Also: I was thinking about windows. fstatat isn't yet implemented on it, from what I see in os.zig. I guess I must keep the old behavior in Dir.statFile for this OS?

[KaneRoot]

I changed the code for Dir.statFile back to what it was for windows. I did a zig build -Dtarget=x86_64-windows and it builds correctly.

Now FreeBSD fails to build, and I don't know what change could possibly result with this:

unable to open '/usr/home/build/zig/zig-cache/o/bec7832f606e3634e2fa939341ffca78/builtin.zig': file not found

Any idea?

The build doesn't fail on my system when I run zig build -Dtarget=x86_64-freebsd.

EDIT: same problem on MacOS, too.

[matu3ba]

Any idea?

Following symlinks has other constants on freebsd and darwin and you unconditionally use the linux ones. For example, AT_SYMLINK_NOFOLLOW is not defined in os/darwin.zig. You can test this on using 0 instead to not follow symlinks.

Darwin uses AT_SYMLINK_NOFOLLOW instead, but those constants are not yet in os/darwin.zig as they are also version-specific. Grepping for those yields in

include/any-macos.11-any/sys/fcntl.h
166:#define AT_SYMLINK_NOFOLLOW     0x0020  /* Act on the symlink itself not the target */
include/any-macos.12-any/sys/fcntl.h
169:#define AT_SYMLINK_NOFOLLOW     0x0020  /* Act on the symlink itself not the target */
include/wasm-wasi-musl/__header_fcntl.h
55:#define AT_SYMLINK_NOFOLLOW (0x1)
include/wasm-wasi-musl/fcntl.h
66:#define AT_SYMLINK_NOFOLLOW 0x100
include/generic-musl/fcntl.h
61:#define AT_SYMLINK_NOFOLLOW 0x100
include/any-linux-any/linux/fcntl.h
98:#define AT_SYMLINK_NOFOLLOW  0x100   /* Do not follow symbolic links.  */
include/x86_64-macos.10-gnu/sys/fcntl.h
139:#define AT_SYMLINK_NOFOLLOW     0x0020  /* Act on the symlink itself not the target */

However, I am not sure if this is the culprit here and if we can and want to maintain the constants here manually.

[KaneRoot]

I fixed the compilation for the different OSs. Symlinks aren't followed for most systems, besides windows, IOS, MacOS and wasi since they don't have SYMLINK_NOFOLLOW.

Is there anything else to change?

[KaneRoot]

I made the requested changes, and build failed on the ziglang.zig platform for the MacOS architecture. MacOS arm64 and Windows architectures worked on ziglang.zig.

I don't see anything useful in the logs. It just spent too much time performing the build and tests, no crash or whatever. I'm a bit confused, my code didn't really change from the last time, everything should be fine. Also, it worked on other platforms and on my machine, for both a Windows and Linux x86_64 builds.

Any idea?

EDIT: I changed a comment and the tests passed. I guess it was a problem with the platform, not with the code.

[andrewrk]

Very strange CI failure. 17 is EEXIST, which, based on reading Linux kernel source code, does not look like it is possible to be returned from fstatat. I also cannot fathom how it is possible for those warnings would be printed, given that Zig atomically renames that file into place.

warning(module): the cached file '/workspace/zig-cache-local-debug/o/f4c284e659e1729c0f0cefdf453c2a53/builtin.zig' had the wrong size. Expected 2877, found 0. Overwriting with correct file contents now
warning(module): the cached file '/workspace/zig-cache-local-debug/o/88f2a22cfc4800a1ebd1ee40f246b369/builtin.zig' had the wrong size. Expected 2239, found 0. Overwriting with correct file contents now
warning(module): the cached file '/workspace/zig-cache-global-debug/o/97aaeaf6369fe56f2629ac6b77c3e687/builtin.zig' had the wrong size. Expected 2245, found 0. Overwriting with correct file contents now
warning(module): the cached file '/workspace/zig-cache-global-debug/o/827e8bb7d561d20acb15d473b3e051b0/builtin.zig' had the wrong size. Expected 2245, found 0. Overwriting with correct file contents now
+ pwd
+ stage3/bin/zig build test -fqemu -fwasmtime -Dstatic-llvm -Dtarget=native-native-musl --search-prefix /deps/local --zig-lib-dir /workspace/build-debug/../lib
warning(module): the cached file '/workspace/zig-cache-local-debug/o/8db843545a30e51b896b59fc2edd16f9/builtin.zig' had the wrong size. Expected 2877, found 0. Overwriting with correct file contents now
warning(module): the cached file '/workspace/zig-cache-local-debug/o/3b80c674af51d68b456fccf88c7877bc/builtin.zig' had the wrong size. Expected 2876, found 0. Overwriting with correct file contents now
unexpected errno: 17
/workspace/lib/std/debug.zig:560:19: 0x60bd6d8 in writeCurrentStackTrace__anon_68705 (zig)
    while (it.next()) |return_address| {
                  ^
/workspace/lib/std/debug.zig:158:67: 0x5e483ce in dumpCurrentStackTrace (zig)
        writeCurrentStackTrace(stderr, debug_info, detectTTYConfig(), start_addr) catch |err| {
                                                                  ^
/workspace/lib/std/os.zig:5468:40: 0x5e42884 in unexpectedErrno (zig)
        std.debug.dumpCurrentStackTrace(null);
                                       ^
/workspace/lib/std/os.zig:4159:45: 0x666ab4f in fstatatZ (zig)
        else => |err| return unexpectedErrno(err),
                                            ^
/workspace/lib/std/os.zig:4111:24: 0x64a5d55 in fstatat (zig)
        return fstatatZ(dirfd, &pathname_c, flags);
                       ^
/workspace/lib/std/fs.zig:2361:42: 0x62fcd8d in statFile (zig)
                const st = try os.fstatat(self.fd, sub_path, 0);
                                         ^
/workspace/src/Module.zig:4012:55: 0x62fc7e6 in populateBuiltinFile (zig)
    if (builtin_pkg.root_src_directory.handle.statFile(builtin_pkg.root_src_path)) |stat| {
                                                      ^
/workspace/src/Compilation.zig:3353:28: 0x62fcf5a in workerUpdateBuiltinZigFile (zig)
    mod.populateBuiltinFile() catch |err| {
                           ^

[matu3ba]

Now the error on x86_64-linux debug looks "better" with INVAL (EINVAL Invalid flag specified in flags. ):

+ stage3/bin/zig fmt --check .. --exclude ../test/cases/ --exclude ../build-debug --exclude ../build-release --exclude /workspace/zig-cache-local-debug --exclude /workspace/zig-cache-global-debug
Looking for non-conforming code formatting...
+ stage3/bin/zig build -Dtarget=arm-linux-musleabihf
warning(module): the cached file '/workspace/zig-cache-local-debug/o/cf3f22d7836595a2edbf42ee81025a33/builtin.zig' had the wrong size. Expected 2920, found 0. Overwriting with correct file contents now
thread 9175 panic: reached unreachable code
/workspace/lib/std/os.zig:4149:19: 0x686cb20 in fstatatZ (zig)
        .INVAL => unreachable,
                  ^
/workspace/lib/std/os.zig:4111:24: 0x66a2225 in fstatat (zig)
        return fstatatZ(dirfd, &pathname_c, flags);
                       ^
/workspace/lib/std/fs.zig:2360:42: 0x64f8a5d in statFile (zig)
                const st = try os.fstatat(self.fd, sub_path, 0);
                                         ^
/workspace/src/Module.zig:4032:55: 0x64f84b6 in populateBuiltinFile (zig)
    if (builtin_pkg.root_src_directory.handle.statFile(builtin_pkg.root_src_path)) |stat| {
                                                      ^
...
???:?:?: 0x7f2ff6343ea6 in ??? (???)
Aborted (core dumped)

On release:

compiler [6/1163] adding_numbers_at_runtime_and_comptime (stage2, aarch64-linux) [1/3] update [... thread 13712 panic: reached unreachable code
/workspace/lib/std/os.zig:4149:19: 0x7ddac20 in fstatatZ (test)
        .INVAL => unreachable,
                  ^
/workspace/lib/std/os.zig:4111:24: 0x7c61c7b in fstatat (test)
        return fstatatZ(dirfd, &pathname_c, flags);
                       ^
/workspace/lib/std/fs.zig:2360:42: 0x7adf85d in statFile (test)
                const st = try os.fstatat(self.fd, sub_path, 0);
                                         ^
/workspace/src/Module.zig:4032:55: 0x7adda36 in populateBuiltinFile (test)
    if (builtin_pkg.root_src_directory.handle.statFile(builtin_pkg.root_src_path)) |stat| {
                                                      ^
...
error: The following command exited with error code 1:
/workspace/_release/staging/bin/zig test --stack 33554432 /workspace/src/test.zig -lc -cflags -std=c++14 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D_GNU_SOURCE -fvisibility-inlines-hidden -fno-exceptions -fno-rtti -Werror=type-limits -Wno-missing-braces -Wno-comment -DNDEBUG=1 -- /workspace/src/zig_llvm.cpp /workspace/src/zig_clang.cpp /workspace/src/zig_llvm-ar.cpp /workspace/src/zig_clang_driver.cpp /workspace/src/zig_clang_cc1_main.cpp /workspace/src/zig_clang_cc1as_main.cpp /workspace/src/windows_sdk.cpp -lclangFrontendTool -lclangCodeGen -lclangFrontend -lclangDriver -lclangSerialization -lclangSema -lclangStaticAnalyzerFrontend -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangAnalysis -lclangASTMatchers -lclangAST -lclangParse -lclangSema -lclangBasic -lclangEdit -lclangLex -lclangARCMigrate -lclangRewriteFrontend -lclangRewrite -lclangCrossTU -lclangIndex -lclangToolingCore -lclangExtractAPI -lclangSupport -llldMinGW -llldELF -llldCOFF -llldWasm -llldMachO -llldCommon -lLLVMWindowsManifest -lLLVMWindowsDriver -lLLVMXRay -lLLVMLibDriver -lLLVMDlltoolDriver -lLLVMCoverage -lLLVMLineEditor -lLLVMXCoreDisassembler -lLLVMXCoreCodeGen -lLLVMXCoreDesc -lLLVMXCoreInfo -lLLVMX86TargetMCA -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Info -lLLVMWebAssemblyDisassembler -lLLVMWebAssemblyAsmParser -lLLVMWebAssemblyCodeGen -lLLVMWebAssemblyDesc -lLLVMWebAssemblyUtils -lLLVMWebAssemblyInfo -lLLVMVEDisassembler -lLLVMVEAsmParser -lLLVMVECodeGen -lLLVMVEDesc -lLLVMVEInfo -lLLVMSystemZDisassembler -lLLVMSystemZAsmParser -lLLVMSystemZCodeGen -lLLVMSystemZDesc -lLLVMSystemZInfo -lLLVMSparcDisassembler -lLLVMSparcAsmParser -lLLVMSparcCodeGen -lLLVMSparcDesc -lLLVMSparcInfo -lLLVMRISCVDisassembler -lLLVMRISCVAsmParser -lLLVMRISCVCodeGen -lLLVMRISCVDesc -lLLVMRISCVInfo -lLLVMPowerPCDisassembler -lLLVMPowerPCAsmParser -lLLVMPowerPCCodeGen -lLLVMPowerPCDesc -lLLVMPowerPCInfo -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMMSP430Disassembler -lLLVMMSP430AsmParser -lLLVMMSP430CodeGen -lLLVMMSP430Desc -lLLVMMSP430Info -lLLVMMipsDisassembler -lLLVMMipsAsmParser -lLLVMMipsCodeGen -lLLVMMipsDesc -lLLVMMipsInfo -lLLVMLanaiDisassembler -lLLVMLanaiCodeGen -lLLVMLanaiAsmParser -lLLVMLanaiDesc -lLLVMLanaiInfo -lLLVMHexagonDisassembler -lLLVMHexagonCodeGen -lLLVMHexagonAsmParser -lLLVMHexagonDesc -lLLVMHexagonInfo -lLLVMBPFDisassembler -lLLVMBPFAsmParser -lLLVMBPFCodeGen -lLLVMBPFDesc -lLLVMBPFInfo -lLLVMAVRDisassembler -lLLVMAVRAsmParser -lLLVMAVRCodeGen -lLLVMAVRDesc -lLLVMAVRInfo -lLLVMARMDisassembler -lLLVMARMAsmParser -lLLVMARMCodeGen -lLLVMARMDesc -lLLVMARMUtils -lLLVMARMInfo -lLLVMAMDGPUTargetMCA -lLLVMAMDGPUDisassembler -lLLVMAMDGPUAsmParser -lLLVMAMDGPUCodeGen -lLLVMAMDGPUDesc -lLLVMAMDGPUUtils -lLLVMAMDGPUInfo -lLLVMAArch64Disassembler -lLLVMAArch64AsmParser -lLLVMAArch64CodeGen -lLLVMAArch64Desc -lLLVMAArch64Utils -lLLVMAArch64Info -lLLVMOrcJIT -lLLVMMCJIT -lLLVMJITLink -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMOrcTargetProcess -lLLVMOrcShared -lLLVMDWP -lLLVMDebugInfoGSYM -lLLVMOption -lLLVMObjectYAML -lLLVMObjCopy -lLLVMMCA -lLLVMMCDisassembler -lLLVMLTO -lLLVMPasses -lLLVMCFGuard -lLLVMCoroutines -lLLVMObjCARCOpts -lLLVMipo -lLLVMVectorize -lLLVMLinker -lLLVMInstrumentation -lLLVMFrontendOpenMP -lLLVMFrontendOpenACC -lLLVMExtensions -lLLVMDWARFLinker -lLLVMGlobalISel -lLLVMMIRParser -lLLVMAsmPrinter -lLLVMSelectionDAG -lLLVMCodeGen -lLLVMIRReader -lLLVMAsmParser -lLLVMInterfaceStub -lLLVMFileCheck -lLLVMFuzzMutate -lLLVMTarget -lLLVMScalarOpts -lLLVMInstCombine -lLLVMAggressiveInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMAnalysis -lLLVMProfileData -lLLVMSymbolize -lLLVMDebugInfoPDB -lLLVMDebugInfoMSF -lLLVMDebugInfoDWARF -lLLVMObject -lLLVMTextAPI -lLLVMMCParser -lLLVMMC -lLLVMDebugInfoCodeView -lLLVMBitReader -lLLVMFuzzerCLI -lLLVMCore -lLLVMRemarks -lLLVMBitstreamReader -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lz -lc++ --cache-dir /workspace/zig-cache --global-cache-dir /root/.cache/zig --name test --pkg-begin test_cases /workspace/test/cases.zig --pkg-end --pkg-begin build_options /workspace/zig-cache/options/fSzd0qh23TW76JKaHuGnkVR8kZeCE4-lxvlfRxetANS8SGcvcIPh-CZFTG_BS3gP --pkg-end -L /deps/local/lib -isystem /deps/local/include --zig-lib-dir /workspace/lib --enable-cache 
error: the following build command failed with exit code 1:
/workspace/zig-cache/o/af52765bc3a7907d7baaa07e4d3004b7/build /workspace/_release/staging/bin/zig /workspace /workspace/zig-cache /root/.cache/zig test docs -fqemu -fwasmtime -Dstatic-llvm -Dtarget=native-native-musl --search-prefix /deps/local --zig-lib-dir /workspace/build-release/../lib