`TCPListener._accept` gets stuck in a busy loop on certain errors

[jemc]

trafficstars

I have observed an issue where TCPListener._accept gets stuck in a busy loop on certain errors, including EMFILE.

I will expound below:

---

On POSIX systems (or more precisely, systems where ifdef windows is false), when we get a "readable" ASIO event notifying us that there are one more connections waiting to be accepted, we enter a loop, accepting until there's nothing left to accept:

https://github.com/ponylang/ponyc/blob/95f38636008a8a54ef0ee95844f46631bdcdc4d9/packages/net/tcp_listener.pony#L207-L220

If you look at the logic of this loop, you can see that it depends on the following assumptions about the return value of the pony_os_accept call:

• a return value of -1 means "no socket; but try again"
• a return value of 0 means "no socket; stop trying to accept" (corresponding to an underlying errno of EWOULDBLOCK)
• any other value is the file descriptor of the new socket

So let's take a look at the implementation of pony_os_accept with that in mind:

https://github.com/ponylang/ponyc/blob/95f38636008a8a54ef0ee95844f46631bdcdc4d9/src/libponyrt/lang/socket.c#L770-L784

We see that on both linux and macos/bsd it will return 0 for an error of type EWOULDBLOCK or EAGAIN, and it will return -1 for any other error type.

The problem with this picture is that there are other error types besides just these two for which an action of "try again" is not a great idea.

The particular one I saw causing a problem in practice was EMFILE, which (along with the similar ENFILE) indicates that a limit has been reached of open file descriptors, meaning no more connections can be accepted at this time (until some of the open file descriptors get closed). In such a situation, the approach of TCPListener._accept will keep looping in a busy loop indefinitely, until the EMFILE condition is cleared.

This particular failure mode can be worked around by either lowering the _limit field of the TCPListener or raising the system limit on the maximum number of open file descriptors. But in the presence of multiple sources of file descriptors, there is no perfect way to guarantee that the system limit won't be reached before the internal _limit field limit.

---

Probably the ideal Pony behavior when receiving an EMFILE is to treat it the same as if the internal _limit had been reached - halt the loop and mark the _paused field as true.

But it also raises questions for me as to whether other failure modes are also being handled correctly. We should look through the full list of possible errors for accept4 on Linux and accept on BSD and accept on MacOS and determine what the proper action is for each of these cases, whether that is:

• try again (currently marked by returning -1 from pony_os_accept)
• don't try again (currently marked by returning 0 from pony_os_accept)
• don't try again and set the listener to be paused, so it can be resumed later (needs a new path to mark this)
• something else?

And then we need to work out whether we should be checking these errno numbers on the C side (in pony_os_accept) or on the Pony side (in TCPListener._accept).

[jemc]

Okay, I've taken a first stab at auditing the possible error types for accept4 on Linux (see the comment after this one).

I'm looking for someone like @SeanTAllen to check and affirm or correct my assumptions. Then once we've agreed on the error codes for Linux I can look at the BSD/MacOS ones (which are likely to be similar).

[jemc]

Retry immediately

These are errors related to an individual connection socket, which if encountered are not expected to happen for all available connection sockets. Hence, we should be able to retry to move past the offending connection socket and get the next available connection socket (if any).

• ECONNABORTED - A connection has been aborted.

• EINTR - The system call was interrupted by a signal that was caught before a valid connection arrived; see signal(7).

• EPERM - Firewall rules forbid connection.

• Linux accept() (and accept4()) passes already-pending network errors on the new socket as an error code from accept(). This behavior differs from other BSD socket implementations. For reliable operation the application should detect the network errors defined for the protocol after accept() and treat them like EAGAIN by retrying. In the case of TCP/IP, these are ENETDOWN, EPROTO, ENOPROTOOPT, EHOSTDOWN, ENONET, EHOSTUNREACH, EOPNOTSUPP, and ENETUNREACH.

• In addition, network errors for the new socket and as defined for the protocol may be returned. Various Linux kernels can return other errors such as ENOSR, ESOCKTNOSUPPORT, EPROTONOSUPPORT, ETIMEDOUT. The value ERESTARTSYS may be seen during a trace.

Done accepting (try again at the next ASIO event)

These errors mean we are out of available connection sockets. We're done accepting. This is the normal, non-exceptional case of just being "all done iterating".

• EAGAIN or EWOULDBLOCK - The socket is marked nonblocking and no connections are present to be accepted. POSIX.1-2001 allows either error to be returned for this case, and does not require these constants to have the same value, so a portable application should check for both possibilities.

Pause (try again at the next ASIO event, or after another connection has closed)

These errors mean we are unable to accept any new connection sockets at this time. The best we can do is mark ourselves as "paused" so we can try to resume later (at the next ASIO event, or after another connection has closed).

• EMFILE - The per-process limit of open file descriptors has been reached.
• ENFILE - The system limit on the total number of open files has been reached.
• ENOBUFS, ENOMEM - Not enough free memory. This often means that the memory allocation is limited by the socket buffer limits, not by the system memory.

"Impossible"

These are conceptually "type system" errors that are expected to be "impossible" to reach because we are calling accept4 correctly with the correct kinds of arguments.

However, because they are not actually covered by a strong type system, we have to account for the possibility somehow that we are wrong in some cases or in some way, so we may encounter these, and if we do, we should at least not do a busy loop (i.e. retry immediately). Instead we should probably treat them the same as the "Pause" errors or the "Done accepting" errors - it doesn't matter much which of those that we choose, I think.

• EBADF - The descriptor is invalid.
• EFAULT - The addr argument is not in a writable part of the user address space.
• EINVAL - (accept4()) invalid value in flags.
• EOPNOTSUPP - The referenced socket is not of type SOCK_STREAM.
• ENOTSOCK - The descriptor references a file, not a socket.
• EINVAL - Socket is not listening for connections, or addrlen is invalid (e.g., is negative).

???

I have no idea at the moment which category this fits in, since it's not very descriptive.

• EPROTO - Protocol error.

[jemc]

For "Impossible" errors we discussed on the sync that we should shut down the listener and do a not_listening event.

For EPROTO we agreed to put it in the "Retry Immediately" camp.

We also got basic consensus on the approach and I'll get started on this fairly soon/

[jemc]

Confirmed that BSD and MacOS don't add any new possible errors, so the plan described above should work for all those platforms.