Prefer depending on NtDll rather than kernel32 or other higher level DLLs

[daurnimator]

trafficstars

Proposal Clarification

---

"windows" as most people think of it is a stack of

• the nt kernel
• ntdll.dll
• kernel32.dll
• user32.dll

You don't have to use all elements of this stack, and there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

The Zig standard library should not assume that kernel32.dll and user32.dll are unconditionally available.

[andrewrk]

there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

What situations are these? Does anyone know?

This is an important proposal if we have some use cases to guide it.

[suirad]

Some background for anyone not familiar: ntdll.dll is the only constant that is required for a windows userland process, as its what does the syscalls. kernel32.dll mostly just wraps ntdll and provides what is colloquially known as Winapi. user32 is essentially the kernel32 of Windows gui stuff. The windows userland stack ends up looking like: winlibc (aka fopen()) -> kernel32 (aka CreateFileA()) -> ntdll (aka NtCreateFile()).

---

A couple examples of when some of these may not be available/wanted:

• In Windows Kernel/Driver development those kernel32.dll, ntdll.dll and user32.dll aren't available.
• In some position independent code that is loaded/unpacked/de-obfuscated from elsewhere

---

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

An example of how this would be immediately useful is in creating std.mutex where windows xp and above has CriticalSections available to use, windows vista+ enables use of SRWLock and windows 8+ has WaitOnAddress available, which is the closest thing to a futex windows has. This would result in the mutex itself resolving which of these would be best to use at compile time rather than at runtime.

[jayschwa]

Windows XP

Microsoft stopped maintaining XP nearly 5 years ago. What is Zig's policy on supporting old operating systems?

[andrewrk]

What is Zig's policy on supporting old operating systems?

I don't think it's relevant to the language. But the standard library does not support Windows XP, since it's not supported by Microsoft.

However anyone is free of course to create a third party package for interfacing with Windows XP. And note that much of the standard library has no OS dependencies, and thanks to lazy analysis, you can therefore use the parts of the standard library that have no OS dependencies even when targeting an unsupported OS.

[PavelVozenilek]

There's one very specific use case, software hardened against tampering, where people "include" these standard DLL's (their code sections) inside the executable.

However, this (and driver development) is something very rare, and there's always present danger of feature creep, under-documentation, untested/untestable functionality.

[andrew-boyarshin]

Proposal

OS

• nt: kernel-mode/drivers
• winnative: native images (like smss.exe, csrss.exe, autochk.exe), user-mode with NTDLL only
• win32: user-mode Win32 subsystem images, with guaranteed existence of NTDLL, KERNEL32 and KERNELBASE (since Win10).
• windows => win32

libc

• msvc: Windows-only
• ros: ReactOS-based
• gnu: MinGW-w64
• wine (with wine-staging)

Compatibility

libc might impose additional dependencies.

• wine - msvcrt depends on KERNEL32, so supported only with win32 OS
• ros - full support
• gnu - not sure about nt and winnative

[andrewrk]

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

This is #1907.

---

I'm accepting this proposal, and to provide further clarification of what such an acceptance looks like, it means:

When the standard library needs to interact with the Windows kernel, if there is ntdll API which provides the necessary components to perform the task correctly, then that is the preferred API to use. Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

Once we shave down the non-NtDll dependencies in the std lib, let's see what we have left and re-evaluate from there.

One strategy that can be used to find this information out is ProcMon. ProcMon can reveal when a higher level DLL is calling into a lower level one.

[daurnimator]

Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

[andrewrk]

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

In some places this is true, for example with futexes. But generally, Zig users have a choice of which layer to target, by choosing whether to link libc. When linking libc, it makes sense to target the libc layer, for a few reasons:

• saving code size in some cases, since the C code must be linked anyway
• necessary for features such as glibc nss to work
• linux users might expect to be able to use LD_PRELOAD to override libc functions. This is how valgrind integrates with std.heap.c_allocator, for example.

However, not targeting the libc layer is also a primary use case, when not linking libc.

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

The Zig project has already managed to get one bug fix upstreamed into Wine. Let's start with the optimistic attitude of improving the open source communities around us, before we give up and make compromises.

Now, if there are reasons to target kernel32 or other higher level DLLs, let's hear those reasons out. If they are compelling enough, then it probably makes sense to add an additional target configuration option, which is available to observe in std.builtin, and the std lib can decide which layer to prefer. But I'm not yet convinced this is desirable.

[adontz]

I am sorry, if I'll sound a bit harsh, but I have found a few false statements in this thread.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits. ntdll is available in kernel mode and user mode, but exports different set of functions for different modes. For kernel mode, i.e. from driver, one will call ZwCreateFile and not NtCreateFile, so exporting user mode ntdll functions for means of driver development is simply a mistake. Such std will not be more useful, useful in more scenarios or something liek that. It's still user-mode only regular application case. Not a kernel driver, not early boot service, nothing like that.

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all. There are no reasons to avoid using kernel32, because it is always available. kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one. Those who link directly to ntdll will have to reimplement all kinds of argument conversion logic, or deal with all kinds of weird behavior.

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change. Drivers will use different (Zw-prefixed) functions anyway. So there are literally no reasons to avoid kernel32.

Also, paths are not compatible and it's not about maximum allowed length or some fancy prefix. "\Device\Harddisk0" is a valid NT path, but not a valid Win32 path with or without any prefix. There are many obvious and even more subtle differences between NT and Win32 behavior, so user mode std depending on NT calls is a really weird choice.

I can't remember a single language standard library (C, rust, D, .Net) to use these functions, and I've dissected many of them.

[andrew-boyarshin]

@adontz

1. Kernel mode can also call Nt* versions. Moreover, it is preferred to when data comes from user mode and cannot be trusted.
2. NTDLL is not used in kernel mode, because NTOSKRNL provides most of the NTDLL's API. Technically there is a common part (RTL) linked into both, difference part (for the function with Nt* name NTDLL usually does syscall, NTOSKRNL actually implements the function), plus a vast number of unique API's in both.
3. KERNEL32 and KERNELBASE are not present in Native Executable image scenario, although it's hard to deny that scenario is rare.
4. About DOS and NT paths: it depends on the function really. Some NTDLL functions expect DOS paths and convert them to NT paths internally before doing syscall, some expect already converted NT path.

Honestly, the proposal I've written might be good, but I've already decided against using Zig for low-level Windows development, so I don't really care if Zig stays user-mode only.

upd: 5. .NET depended on a lot of internal functions around the 3.0-3.5 timeframe. Later they migrated to stable APIs, or, where applicable, just started to bring the code from Windows into their own and adapting to their needs.

[daurnimator]

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all

Ah, appears they added that in XP for the windows subsystem. Most of the books on windows internals are from the Win 2000 era :( Still, writing native subsystem programs is useful and interesting. midipix has a few examples.

kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one

That's sort of the issue: lots of kernel32 wrappers don't expose the full functionality of the NT API. From the basic stuff like missing flags, to different concepts of paths (nt paths are not null terminated, but length delimited resulting in end users needing tools like RegDelNull) to just plain missing functions (e.g. did you know nt supports fork... but kernel32 doesn't).

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change.

There have only been 2 notable function changes I know of since windows 2000, and they have coincided with major releases: despite Microsoft's statements, they are quite stable. What is unstable is the actual syscall numbers, which change even between monthly windows updates.

[andrewrk]

@adontz, I appreciate your input.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits.

There is one very real benefit: the APIs are more powerful. For example, NtCreateFile has the ability to open a sub-directory path based on an open directory handle. The kernel32 APIs do not expose this. It has meaningful consequences in terms of avoiding file system race conditions.

When I first started working on Windows implementations of cross platform abstractions, I took the Microsoft documentation very seriously. But then I found out that pretty much every standard library uses SystemFunction036 from advapi32.dll for getting cryptographically secure random bytes, while Microsoft's docs says to use CryptGenRandom. The docs for CryptGenRandom now say

Important This API is deprecated. New and existing software should start using Cryptography Next Generation APIs. Microsoft may remove this API in future releases.

What I learned is that it's more important to consider ABIs rather than APIs.

Microsoft won't break RtlGenRandom and they won't break NtCreateFile. If they do, they have a lot more programs to worry about breaking than Zig binaries.

The bottom line here is that NtDll is lower-level ABIs to the kernel, and kernel32 is higher level code that wraps NtDll. This is clear when you look at calls in ProcMon. And frankly, Zig's open-source code that wraps NtDll is more robust and reliable than Microsoft's cruft inside kernel32.dll. I don't remember the specifics, but off the top of my head, there are functions in kernel32 that wrap NtDll functions that allocate heap memory, whereas the zig code that wraps NtDll functions does not. That can be the difference between deterministic latency or nondeterministic latency.

At this point I'm convinced that NtDll is an entirely appropriate abstraction layer for windows applications to target. My mind is open to counter-examples, but calls to NtDll functions have been in the zig std lib for quite some time now, and I'm not aware of a single issue it has caused. On the other hand, it has allowed us to unify the std.fs API across POSIX and Windows, since with NtCreateFile they both support operating on an open directory handle.

@andrew-boyarshin,

I've already decided against using Zig for low-level Windows development

Is there anything the Zig project can learn from your decision?

[andrew-boyarshin]

@andrewrk note that the following is purely about Windows kernel drivers, ReactOS kernel and NTDLL, user-mode Zig is cool, I play with it (but yet to use it in any real project).

1. I'm not sure Zig would allow me to develop kernel driver code (standard library)
2. I'm not sure Zig would allow me to throw and handle SEH exceptions properly (ExceptionCode, user buffer probing, exception filters)
3. I'm not sure about calling conventions being compatible, but that might well be false. I've never tried Clang/LLVM for low-level code, and Zig is 2 layers where mistakes can exist. Clang+LLVM is 1.5 layer (Clang is very coupled with LLVM, so it's difficult to count them separately). I just feel MSVC is a safer (as in error-proof) bet (single layer monolith used by Windows team themselves).
4. I'm not sure I can rely on Zig features being safe (as in secure) and fast (as fast as they can theoretically be when implemented in C using kernel APIs to the fullest). I'm sure writing C-like Zig will be ok, but when it comes to coroutines and complete kernel asynchrony (can they survive preemption? how will they play with IRQL?)...

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

upd: 5. Status codes. Zig has an awesome concept of static errors and obligatory explicit handling. Unfortunately it's impossible to use that since even MS Docs don't specify all possible NTSTATUS codes, not to mention non-documented functions. So, Zig offers no advantage in this respect.

[daurnimator]

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

Is this something you'd like to take on @andrew-boyarshin ?

[andrew-boyarshin]

@daurnimator not in the foreseeable future. Certainly not in the next 5 months.

[adontz]

There is one very real benefit: the APIs are more powerful. For example, NtCreateFile has the ability to open a sub-directory path based on an open directory handle. The kernel32 APIs do not expose this. It has meaningful consequences in terms of avoiding file system race conditions.

Do you assume that directory cannot be deleted and/or moved while there is an open handle?

When I first started working on Windows implementations of cross platform abstractions, I took the Microsoft documentation very seriously. But then I found out that pretty much every standard library uses SystemFunction036 from advapi32.dll for getting cryptographically secure random bytes, while Microsoft's docs says to use CryptGenRandom.

I see no mention of cryptographic security of this function. But I do not expect some std.random to be cryptographically secure. I think reasonable expectations are that std.random should have nice distribution and be fast, but not crypto secure.

On the other hand, it has allowed us to unify the std.fs API across POSIX and Windows, since with NtCreateFile they both support operating on an open directory handle.

This is a strong argument. Really, I like it.

However, converting paths between namespaces is not an easy task. You may convert "C:\Windows\System32" to some "??\C:\Windows\System32" and go away with that, but what about the opposite direction? "\SystemRoot\System32" may be "C:\Windows\system32", "D:\Windows\system32" and so forth up to "Z:\Windows\system32". Windows installations are automated in enterprise environment and you can install on any large enough properly formatted partition of any disk and assign any unused drive letter during setup you cannot just assume C:

https://blogs.msdn.microsoft.com/jeremykuhne/2016/05/02/dos-to-nt-a-paths-journey/ https://stackoverflow.com/questions/4445108/how-can-i-convert-a-native-nt-pathname-into-a-win32-path-name

As far as I know there is not official way of converting NT path to Win32 path.

[daurnimator]

As far as I know there is not official way of converting NT path to Win32 path.

Why would you need to do so? Note that some NT paths are impossible to represent as a win32 path.

@adontz you might find https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html?m=1 interesting.

[adontz]

@daurnimator I know of RtlDosPathNameToRelativeNtPathName_U, it was never documented.

Why? To convert paths back. As far as I remember NtQueryInformationProcess returns NT path, but GetProcessImageFileNameW does not.

Note that some NT paths are impossible to represent as a win32 path.

Exactly my point. I don't know what is your experience, but I'd rather stay away of these vague path conversion rules.

I think our conversation smoothly shifted from Win32 vs NT, to documented vs undocumented. There are a few attributes I want to focus, if you let me to.

1. Documented / Undocumented.
2. Supported / Deprecated (may affect security, compatibility).
3. Available in all OS versions / Available in some (maybe even not latest) OS versions.

I personally prefer documented, not deprecated APIs, with safe fallback logic for APIs not available on some OS versions. For instance, I hardly imagine any reasonable fallback for TaskDialog or HTTP Server API.

Windows 7 extended support just expired (January this year). I think Windows 8.1 is a safe bet.

[daurnimator]

Why? To convert paths back.

When would you do that? For display purposes?

[adontz]

Why? To convert paths back. When would you do that? For display purposes?

1. Yes, display purposes are important. If you cannot find a file or have any other file related problem, logging or displaying \Device\ paths will make all users and IT support freak out.

2. Not only display purposes. NtQueryInformationProcess returns path like this

"\Device\HarddiskVolume5\Users\Adontz\Projects\VS2019\NtPathTest1\Debug\NtPathTest1.exe"

Now, imagine, I want to load a vendored dynamic library, or try to load a vendored dynamic library. It may be any closed source binary I want to use, or even precompiled binary distribution of open source. To avoid all kinds of errors I would like to give LoadLibraryW an absolute path. Of course it returns NULL and I get error 126 "The specified module could not be found.", because LoadLibraryW is Win32 and has no idea where is "\Device\HarddiskVolume5".

[AlexKotik]

I can see one big downside about using ntdll based apis instead of kernel32 based: users that will compile stdlib code with ntdll apis will get all kind of antivirus false positives. As using ntdll apis is not common for user land programs, but quite common for different kinds of malware.

[GavinRay97]

Maybe this isn't the best place to ask this, but this seems close enough.

This happens when trying to link and call a 70-line Zig staticlib that exports a single method from C:

$ clang-cl test.c && ./test.exe
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtClose referenced in function std.os.windows.CloseHandle
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtCreateFile referenced in function std.fmt.formatText.88
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtWaitForKeyedEvent referenced in function std.fmt.formatText.125
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtCreateKeyedEvent referenced in function std.math.absCast.146
test.exe : fatal error LNK1120: 4 unresolved externals

The only way that this doesn't happen is if the library is built with -Drelease-small or -Drelease-fast. Building with -Drelease-safe still causes the issue, but this time the referenced functions are different:

$ clang-cl test.c && ./test.exe
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtCreateKeyedEvent referenced in function std.array_list.ArrayListAligned(u8,null).appendSlice   
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtWaitForKeyedEvent referenced in function std.array_list.ArrayListAligned(u8,null).appendSlice  
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtClose referenced in function std.mem.Allocator.free.28
rpp_parser.lib(rpp_parser.obj) : error LNK2019: unresolved external symbol NtCreateFile referenced in function std.unicode.utf8Decode
test.exe : fatal error LNK1120: 4 unresolved externals
clang-cl: error: linker command failed with exit code 1120 (use -v to see invocation)

Now, I have read a good chunk of this thread and I now understand that the inclusion of ntdll is nearly required on Windows, but that leaves me with a couple of questions.

• How do I build a static lib on Windows without external dependencies -- what limitations and flags do I need to adhere to?
• Why is ntdll being included, is it part of Zig core functions that being pulled in under the hood when I compile?

If you search the Discord history, you will find similar questions/confusion. Here another user in 2020 is confused why ntdll is being linked for a 20 line itoa() implementation that imports nothing:

• https://discord.com/channels/605571803288698900/606246764651085829/712749168262643785
• https://github.com/DutchGhost/zigiffy/blob/07341af2aa7e6985aab18e9ccdd3909417c99003/zig/src/zig.zig#L23-L57

Also please bear with me if these are pants-on-head dumb questions, I really don't know much about low level development.

[daurnimator]

@GavinRay97 on windows, all programs get ntdll loaded into them automatically when they start: its impossible to opt-out of it, even for "static" applications/libraries. Windows has no stable syscall interface, you instead call all syscalls via their ntdll wrappers.

[GavinRay97]

Windows has no stable syscall interface, you instead call all syscalls via their ntdll wrappers.

Good lord. Alright haha, I never should have left Ubuntu. Thank you for the answer =)

So does that mean that static libs on Windows aren't really a "thing"?

---

EDIT: (For what it's worth, my .lib seems to be working fine from C in -Drelease-fast and -Drelease-small mode, without ntdll)

[MarcTCruz]

However, converting paths between namespaces is not an easy task. You may convert "C:\Windows\System32" to some "??\C:\Windows\System32" and go away with that, but what about the opposite direction? "\SystemRoot\System32" may be "C:\Windows\system32", "D:\Windows\system32" and so forth up to "Z:\Windows\system32". Windows installations are automated in enterprise environment and you can install on any large enough properly formatted partition of any disk and assign any unused drive letter during setup you cannot just assume C:

It's possible to get absolute path from process, on windows, with GetModuleFileName, so, when the root is missing as in your example, process root is taken instead.

[matu3ba]

Ntdll compatibility list taken from https://www.geoffchappell.com/studies/windows/win32/ntdll/api/index.htm "The very large table on this page lists all the functions and variables—there are well over two and a half thousand—that appear in the export directory of any known i386 (x86), amd64 (x64) or wow64 build of NTDLL.DLL from Windows NT up to and including the original Windows 10."

[matu3ba]

Counter-argument for relying on it for accurate runtime process or thread info from https://learn.microsoft.com/en-us/answers/questions/1183193/totalprocessortime-differs-between-windows-10-and also mentioned in https://github.com/ziglang/zig/issues/5191#issuecomment-1661030019 and https://github.com/ziglang/zig/pull/16638#discussion_r1280944719:

"Microsoft broke backwards compatibility and GetSystemInfo, GetThreadTimes, NtQuerySystemInformation and quite a few other functions are busted and broken on Windows 11 when you have more than 1 processor group.

Starting with Windows 11, process and thread affinities span all processors in the system, across all processor groups by default. Each processor group has completely different processor configuration and statistics... GetSystemInfo, GetThreadTimes and NtQuerySystemInformation only return values specific to the current thread' processor group affinity and they don't query multiple processor groups.

The Windows 11 kernel scheduler will execute your process/thread on different processor groups while you're calling those functions and this causes those functions to return completely different set of values that cannot be compared. Every program using those functions currently shows incorrect information exactly like what you've mentioned in your post.

You either need to call SetThreadGroupAffinity and restrict the thread to a specific group so those functions don't switch group and return incorrect values (and ignore statistics for other processors) or update your code and query the statistics for all processor groups by calling NtQuerySystemInformationEx."

However, standard things like filesystem and always loaded things should be fine.

[Jarred-Sumner]

I'm about one week in to porting Bun to work on Windows and so far using std.os is pretty rough

• Invalid file path? That's a segfault due to the use of unreachable
• std.os.access crashes with a file name that doesn't exist
• handful of methods have incorrect type definitions for the W versions
• the buffer passed to NtCreateFile is not supposed to be exactly sized since reparse points exist
• Can't use stat(), so you have to use the more specific functions in std.fs
• zig uses the mingw Abi instead of msvc by default which causes missing symbol errors that were very difficult to narrow down

[andrewrk]

Please either submit bug reports, or don't. The above comment is just noise.