support select.epoll?

[jamadden]

Is this possible in a way that's close enough and performant enough? I'm not sure it is, but it has come up before. Not all libraries check to see if select.epoll exists before they try to call it. OTOH, maybe the better thing to do would be to submit PRs to such libraries to use selectors which handles the fallback automatically.

See https://github.com/benoitc/gunicorn/issues/1494

[sandys]

+1 this is important for us as well. Lots of libraries like Pusher.com breaks down with gevent because of this select.epoll problem. https://github.com/pusher/pusher-http-python/issues/88

We use python 3.6

[oldani]

I was wondering if maybe support for select.kqueue could be added too.

[jamadden]

I think select.kqueue is even more unlikely than select.epoll because it exposes an even richer interface---for example, it can be used to watch for process events where epoll is still restricted to file descriptors. It might be possible to emulate some of what it does with the primitives provided by libev, but certainly not all of it.

Which is the same problem that epoll has. We can emulate some of it, but not all of it. libev has nothing equivalent to EPOLLET (but that's probably not a big deal for most uses), it doesn't have EPOLLONESHOT (but we can probably emulate that), it doesn't have EPOLLEXCLUSIVE (and that one may be tricky to emulate but it might just be possible) and it doesn't have EPOLLRDHUP, EPOLLHUP, EPOLLPRI, EPOLLRDBAND, or EPOLLWRBAND (all of which are difficult or impossible to emulate, but I'm guessing are less frequently used---then again, if you're specifically using epoll instead of the platform-default selector, you may want those features).

Now, instead of trying to emulate these on top of libev, we could use the threadpool and direct the blocking calls off to another operating-system thread, similar to what we do with DNS. But that has a performance cost, and these functions are often used in performance-sensitive areas of code.

Still, maybe having a slow version is better than no version at all? Maybe, but only if the application requires epoll and doesn't have a fallback from epoll to select (which we implement natively)---if it does, we just made things worse for no good reason. And in the case of kqueue it's possible that there would be some conflicts combining its ability to watch processes with libev/gevent's child watchers.

[thedrow]

I'm confused. libev has at least some epoll support through https://github.com/enki/libev/blob/master/ev_epoll.c Couldn't we implement the rest or raise an error if something isn't supported? I'd rather have partial support and declare it as such with a warning raised on each invocation.

[jamadden]

ev_epoll.c is not part of the libev API. Those are implementation functions subject to change at any moment. Moreover, the important ones are static, meaning they won't be available if we're dynamically linked to libev. Even if none of that were true, libev still simply doesn't support all the options that epoll does---they're just not accessible with the libev objects: the constants are not defined.

So basically, yeah, we do use epoll through libev on the appropriate platforms. But we're already taking exactly as much advantage of it as libev exposes. (libuv exposes even less.) Exposing anything more is either some level of a partial emulation, or a threadpool call.

[lambdaq]

Sorry but I have a question to ask. Is it possible to handle server closed connection event in Gevent? I guess in epoll there's a EPOLLHUP can be captured in application layer, but @jamadden suggested that Gvent can not receive such event. Is there any workaround?

[thedrow]

Sorry but I have a question to ask. Is it possible to handle server closed connection event in Gevent? I guess in epoll there's a EPOLLHUP can be captured in application layer, but @jamadden suggested that Gvent can not receive such event. Is there any workaround?

You should probably create another issue for this question.