Make network I/O interruptible on Windows

[joeyadams]

Currently, for Windows with the threaded RTS, network I/O is done with blocking FFI calls. Since FFI calls can't be interrupted, network I/O can't be interrupted. The following example illustrates the problem:

-- Compile on Windows with -threaded
import Control.Concurrent
import Network
import System.IO
import System.Timeout

server sock = do
    (h, host, port) <- accept sock
    putStrLn $ "server: accepted connection from " ++ host ++ ":" ++ show port
    threadDelay 20000000 -- Make client recv block for a long time
    hPutStrLn h "Thank you for your patience"

client = do
    h <- connectTo "localhost" $ PortNumber 1234
    putStrLn "client: connected to server"
    m <- timeout 5000000 $ hGetLine h
    case m of
        Nothing   -> putStrLn "client: hGetLine took too long"
        Just line -> putStrLn $ "client: received " ++ show line

main = withSocketsDo $ do
    sock <- listenOn $ PortNumber 1234
    _ <- forkIO $ server sock
    client

Here, timeout doesn't really work. It does eventually time out, but only after the server finally sends data, waking up the blocking system call.

If you compile on Windows without -threaded, the timeout does work, but the operation isn't actually canceled.

This is a real problem for networked programs that need to run unattended for long periods of time. If a host becomes unresponsive, it can cause unexpected blockage. Working around this problem is difficult and annoying.

---

Windows does not seem to have a scalable socket polling facility like epoll or kqueue. Here are some of our options:

• I/O completion ports
• WSAEventSelect / RegisterWaitForSingleObject
• select

From what I've read, I/O completion ports is the most efficient and scalable concurrent I/O facility on Windows. However, some problems make it difficult to use:

• Need to use system functions that support overlapped I/O, such as ConnectEx, AcceptEx, WSARecv, etc. Some of these functions have hidden surprises. For example, ConnectEx requires the socket to be initially bound, and requires setting the SO_UPDATE_CONNECT_CONTEXT socket option before shutdown will work properly.

• Overlapped I/O is sensitive to the calling thread:

  • When a thread exits, all pending overlapped I/O issued by that thread is canceled.
  • CancelIo cancels all pending I/O issued by the calling thread for a given handle. CancelIoEx lets you specify the operation to cancel, but it requires Windows Vista or later.

  It is possible to work around this by sending I/O requests to worker threads. However, this means two thread context switches per operation. To avoid this, we'd need to integrate completion port handling into GHC's scheduler.

WSAEventSelect, in tandem with RegisterWaitForSingleObject, can be used to poll sockets, and scales up to roughly 25,000 concurrent waits. The downside is that WSAEventSelect automatically puts the socket in nonblocking mode, and cancels any other WSAEventSelect operations on the same socket. This means we would need to coordinate the waits.

select is the least scalable option, but is easiest to implement. The issue with select is that it can't be interrupted asynchronously. Potential workarounds:

• Call select over and over with a timeout.
• Create a dummy socket, pass it to select, and close the socket to wake up the select call. This is a little dangerous, though. If we close the socket before select begins, another thread may allocate a socket with the newly-freed descriptor number.

---

Here's what I plan to do:

• Switch to nonblocking sockets on Windows. Poll by calling select, from a forkIO, again and again with a timeout.
• Use the new polling facility throughout the Network.Socket API (most of the preliminary work has already been done).
• Implement our own IODevice and BufferedIO instances in the network package, at least for Windows.

Using select isn't the most efficient solution, but it's the easiest to implement. I currently don't have the time or the need to implement the I/O completion port approach.

[tibbe]

• How does this related to the work you've been doing creating an I/O manager for Windows? Will the Windows I/O manager eventually have threadWaitRead?
• Would you be willing to maintain the Windows code in the network package. I've limited ability to work on it as my only Windows setup is a barely usable VM that I use for basic testing before releases?

[joeyadams]

How does this related to the work you've been doing creating an I/O manager for Windows?

I want to postpone the new I/O manager for Windows (or at least its use in the network package), for a few reasons:

• We'd need to associate every new socket with the I/O manager's completion port. This means that any application that creates sockets using the MkSocket data constructor will break.
• It's more work to use overlapped variants of all the socket functions, and more testing to make sure they all work as expected.
• To avoid two OS thread context switches per operation (quite expensive, ~20μs on my machine), the new I/O manager will need to be integrated into the scheduler.
• I'm not getting enough feedback on my GHC Trac ticket (#7353).

Will the Windows I/O manager eventually have threadWaitRead?

Winsock doesn't seem to have a polling function that supports overlapped I/O, so probably not. However, calling WSARecv or WSASend with length zero might work. I haven't tried it, though.

We'll have an API like this instead:

-- | Identifies an I/O operation.  Used as the @LPOVERLAPPED@ parameter
-- for overlapped I/O functions (e.g. @ReadFile@, @WSASend@).
newtype Overlapped = Overlapped (Ptr ())

-- | Must be called once on every handle that will be used with 'withOverlapped'.
-- If omitted, 'withOverlapped' will hang uninterruptibly.
--
-- The handle is automatically dissociated when closed.
associateHandle :: HANDLE -> IO ()

-- | Start an overlapped I/O operation, and wait for its completion.  If
-- 'withOverlapped' is interrupted by an asynchronous exception, the operation
-- will be canceled using @CancelIo@.
withOverlapped :: HANDLE -> (Overlapped -> IO ()) -> IO Completion

-- | Returned when the completion is delivered.
data Completion = Completion
    { cErrCode  :: !ErrCode -- ^ 0 indicates success
    , cNumBytes :: !DWORD   -- ^ Number of bytes transferred
    }

Would you be willing to maintain the Windows code in the network package. I've limited ability to work on it as my only Windows setup is a barely usable VM that I use for basic testing before releases?

I suppose so. I bought a copy of Windows 8 because it was cheap (offer ends Jan 31, 2013), and made it my primary system (for now) so I can fix this issue.

[tibbe]

If you feel comfortable making the changes to network to make it non-blocking on Windows and owning the result (e.g. fixing any bugs that crop up) I'm happy taking the patches.

[kazu-yamamoto]

@Mistuke Would you give a look at this issue and close it if possible or if too old.

[kazu-yamamoto]

This is out dated. Let's close.