swift-nio
swift-nio copied to clipboard
apple
POC output with URring
Use URing on Linux.
URing on Linux provides an API to demultiplex events and perform socket operations. NIO originally designed to demultiplex events and perform all IO in a single thread, while URing socket operations are asynchronous. It requires more significant changes in the NIO taking into account the requirement to additionally manage memory buffers for pending read and write operations.
The one idea was to try to minimise PR as possible, so that PR send outbound data using URing, but still read inbound data using base socket API. Reading inbound data with URing is a next step.
Some highlights regarding changes in the PR
-
The URing API using on Linux can be enabled by the conditional compilation. In some places I saw '#if defined(SWIFTNIO_USE_IO_URING) && (os(Linux) || os(Android)) As of now we have no possibility even to compile it for Android, not saying to test, so all URing related code is under just os(Linux). Let's decide later what can we do with Android.
-
NIO used some buffers stored in the EventLoop (iovecs, storageRefs etc) which are shared across all channels registered in the event loop. That approach does not work with URing because of its asynchronous nature. There are not too many ways to workaround it for URing, so I moved them to the channel, so with URing each channel has own iovecs and storageRefs buffers. Probably we could cache them... not clear what is better... let's discuss...
-
URing has no analogue for the sendfile() system call. Instead they suggest to splice() the file via intermediate pipe. It works, but requires an intermediate pipe. There are not too many options how to manage intermediate pipes, but I can't see a good one. So, as of now we just cache intermediate pipes and reuse them...
-
URing has no analogue for sendmmsg() system call. Unfortunately did not found a good way to workaround that. Currently just send messages one by one. One possible optimisation here is to collect all messages with the same destination address and same ancillary data and send them with one sendmsg(), but did not found such obvious optimisation in the NIO, so probably it is not a case.
-
I had to disable few tests which checks the order of system calls. Have no idea how we could make them work for URing, because URing is asynchronous. Let's discuss them later. All other tests works on my environment, but I will not be surprised if they will not work somewhere else.
Will add some information later within comments to PR if will recall something important.
I got the point regarding the conditional compilation. How do you think should we move the conditional compilation in that place more deeply leaving the async API available even for a case if the backend is synchronous?
I'm inclined to suggest that we have most backends implement the functions with fatalError, and only have the uring selector implement them. Then we can ensure that at runtime we only call those functions when we know we have the uring selector, which we should know either as a compile-time static or as event loop state.
Could you have a look at PR? Is it appropriate way to remove conditional compilation in that particular case from your point of view?
Could you have a look at PR? I tried to disable possibility to send files with URing, but end up with a lot of conditional compilation in tests. As far as I understand test runner parse swift code and generate a code calling tests, so I have no better idea how could disable those tests when running with URing. What do you think?
I'm entirely happy with us adding this conditional compilation into the tests, as it's small and clear. :+1:
I merged most recent changes from the main into the feature branch and stabilised tests, so all tests (except skipped are passed now). Had a lot of fun with the test stabilisation, it still seems to me the idea to have both SYNC and ASYNC code at the same time in the source code is not good and it would be much better separate them with #ifdefs. The point here is that on runtime the only one approach should work, but codebase has a lot of conditional branches, and sometimes it happen that SYNC IO is triggered when ASYNC is supposed to because source code was not fixed properly. It would not even happen if we would completely move the SYNC functionality under #ifdef. Not sure I found all places where SYNC IO can be triggered in ASYNC mode. But still, all tests pass now, would be nice if you could have a look at PR.