cqueues
cqueues copied to clipboard
wahern
Windows support
I've been mulling over how hard adding windows support to cqueues would be. Lower level windows programming is not something I'm familar with, so please correct any incorrect assumptions I've made.
I'd love to have cqueues be a library I can depend on everywhere: if I write a network client in cqueues, even windows developers should still be able to use it (though maybe not in a performant way, which is the approach many applications take, e.g. nginx).
Comments @wahern has made before:
The concept is not portable to Windows because Windows lacks an analog to the pollable event queues of modern Unix systems. Libraries which try to abstract the two approaches—event readiness signaling versus event completion—invariably produce a pallid least common denominator library suffering all the weaknesses and enjoying none of the strengths of each approach.
Microsoft Windows support is basically out of the question1, for far too many reasons to put here. Aside from the more technical reasons, Windows I/O and networking programming interfaces have a fundamentally different character than on Unix. Unix historically relies on readiness polling, while Windows uses event completion callbacks. There are strengths and weaknesses to each approach. Trying to paper over the chasm between the two approaches invariably results in a framework with the strengths of neither and the weaknesses of both. The purpose of cqueues is to leverage the strengths of polling as well as address the weaknesses.
1 I have been toying with the idea of using an fd set in-place of a pollable descriptor on Windows, and taking the union of all fd sets when polling.
Solution 1. Add :pollset
One possible route is catering to the lowest common denominator (which helps with other platforms, such as AIX) and use select(). This can be accomplished by iterating over the fileno LLRB and building a pollset; this could be done on demand in a :pollset() function, or inside of fileno_ctl.
Issues
select does not expose a file handle we can wait on itself, and makes things much harder to nest/stack, :pollsets will need to be propogated up to the parent controller.
On windows, select boils down to a WaitForMultipleObjectsEx call, this has a maximum of 64 handles; which could easily be exceeded by a regular application. To get around this, you can spawn extra threads that each wait for events, and wait for them, and now we've recreated stackable event waiting.
Solution 2. Use an IOCP and RegisterWaitForSingleObject
- Create an IOCP port for each cqueues object.
- To 'add' a file descriptor to it, use
RegisterWaitForSingleObjectwhere the callback usesPostQueuedCompletionStatus - To cancel a file descriptor, use UnregisterWaitEx
kpoll_waitusesGetQueuedCompletionStatusEx:pollfd()returns the IOCP handle
Issues
Windows potentially has more than just readable and writable events; you need to use WSACreateEvent + WSAEventSelect to manage a HANDLE that encapsulates the events being waited for.
It is not possible to specify different event objects for different network events. The following code will not work; the second call will cancel the effects of the first, and only the FD_WRITE network event will be associated with hEventObject2:
rc = WSAEventSelect(s, hEventObject1, FD_READ); rc = WSAEventSelect(s, hEventObject2, FD_WRITE); //bad
Solution 2.5:
@piscisaureus alerted me to the undocumented IOCTL_AFD_POLL fast polling interface.
This is what select() on windows uses internally, it's not limited to 64 descriptors.
libuv uses it.
This is apparently incompatible with some LSPs, but they're deprecated, so we can probably just not support that case.
On Thu, Mar 26, 2015 at 05:34:46PM -0700, daurnimator wrote:
Solution 2.5:
@piscisaureus alerted me to the undocumented
IOCTL_AFD_POLLfast polling interface. This is whatselect()on windows uses internally, it's not limited to 64 descriptors. libuv uses it. This is apparently incompatible with some LSPs, but they're deprecated, so we can probably just not support that case.
Very cool. Looks like the work originated here
https://github.com/piscisaureus/epoll_windows
according to this thread
http://comments.gmane.org/gmane.comp.lang.javascript.nodejs.libuv/1049
More information about what AFD is: http://mista.nu/blog/2012/02/17/cve-2012-0148-a-deep-dive-into-afd/
I had a play around with AFD today, check out this example: https://gist.github.com/daurnimator/63d2970aedc952f0beb3
One of the roadblocks I learnt about was that you can't have an IOCP listen for events on an IOCP. Or at least, I haven't figured it out yet.
- You can't add an IOCP handle to another IOCP
- You can't register waits on IOCP handles
Maybe we can have some arrangement where a worker thread polls GetQueuedCompletionStatus and posts across? I'm not sure...
For the pollfd code style to work, it needs to be able to return any type of HANDLE, I imagine we have users yield HANDLEs as lightuserdata?
Once you have a handle you may need to know what to do with it. To find out the type of a HANDLE, you need to use NtQueryObject() with PUBLIC_OBJECT_TYPE_INFORMATION, and dispatch based on TypeName.
My current opinion has tended back to Solution 1.
It seems that an array of HANDLEs will be needed no matter what: the windows primitive of WaitForMultipleObjectsEx and the AFD stuff I found both need it.
If we implement :pollset now, we can then special case socket descriptors later to go via AFD.
Solution 1. Add :pollset
I played around with adding a :pollset function; + a pair'd function :pollresult over on a branch: https://github.com/daurnimator/cqueues/tree/pollset
@katlogic suggested using https://github.com/richardhundt/upoll But it looks like upoll doesn't implement "stacking"; however it might not be hard to add.
On Mon, Mar 07, 2016 at 06:31:51PM -0800, daurnimator wrote:
@katlogic suggested using https://github.com/richardhundt/upoll But it looks like upoll doesn't implement "stacking"; however it might not be hard to add.
It's not doing anything that the existing code isn't already doing except for a dozen or so lines for select. It'd be easier to add select to the existing code than to add kqueue, ports, etc support to upoll.c. Likewise for adding select stacking.
I had an idea for supporting Windows overlapped I/O in a performant manner, but it first involves properly breaking out the buffering code.
But simply supporting select on Windows is probably the best interim measure. Slightly easier (stacking notwithstanding) would be just supporting WSAPoll, available since Windows 2008.
In midipix there's an interesting approach: for sockets, use AFD; but for other handle types, do a 0 byte read/write using the NT API in a worker thread. These block until there is data available for read/write. At which point you can signal an event. This works out at scale because on NT, canceling io operations is cheap.
On Thu, May 19, 2016 at 09:46:19PM -0700, daurnimator wrote:
In midipix there's an interesting approach: for sockets, use AFD; but for other handle types, do a 0 byte read/write using the NT API in a worker thread. These block until there is data available for read/write. At which point you can signal an event. This works out at scale because on NT, canceling io operations is cheap.
Can you post a link?
I suggest use select on windows. Generally in windows are use for client. like https://github.com/daurnimator/lua-http , I want use http client in windows, But now can't.
Here has a epoll in windows by select.
https://github.com/dpull/skynet-mingw/blob/master/platform/epoll.c
Somewhat tangential, but I was reading up on Linux's io_uring API and had the thought that adding support for it in cqueues might make it simpler to also support IOCP, given that there seem to be similarities.
I'm oversimplifying and understand that this would likely be a lot of work. I suggest this because cqueues and Lua-HTTP are my favorite libraries for their purpose and a lack of Windows is a frequent issue.
Looking in the comments, I see that @wahern isn't huge fan of this, so maybe it's not a great fit. :)