uvloop
uvloop copied to clipboard
MagicStack
windows support?
@1st1 Hi, I'm interested in implementing this proposal, is anyone already working on it ;)?
@1st1 I have a little trouble porting posix.signal to Windows in https://github.com/MagicStack/uvloop/blob/master/uvloop/os_signal.pyx#L1 . I could not find a good way to implement it because of the difference in signal between Windows and *nix. Could you please give me some advice?
I don't think Windows supports signals, so you should probably add checks like if not __WINDOWS__ around code that touches loop.py_signals and loop.uv_signals.
@1st1 Sure, I find that there isn't a quick way to port uvloop with the native Windows API. I would try to port uvloop with Cygwin these days first, cuz I find Cygwin provide most of the *NIX API on Windows.
@1st1 I find that Cygwin haven't provided Python3.5 officially :(, so we should work on it with the native Windows API again. Maybe we should find more people to look into this problem.
There are two major parts we need to port, socket and signal.
For socket, most of socket functions could be easy to ported by changing the header file, but Windows don't provide socketpair. Referring to https://github.com/ncm/selectable-socketpair and https://github.com/python/cpython/blob/c6ae2fcc724cbebb14e7c434b89eabdd64802cb3/Lib/asyncio/windows_utils.py#L37, it's possible to implement a socketpair that support AF_INET and AF_INET6, but it doesn't support AF_UNIX.
For signal, Windows provides a function signal(https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx) to change the handlers of specific signal, but it doesn't provide the function to get the current handler of specific signal and it also doesn't provide sigfillset.
Libuv already provides a platform agnostic interface to signals and sockets (e.g. http://docs.libuv.org/en/v1.x/signal.html). You shouldn't have to deal with the Windows API at all.
@schlamar You should have looked at uvloop source code first, before assuming that we don't know some basic libuv APIs.
@brickgao Sorry for the delayed reply. For signals specifically I use sigaction to make sure that:
- uvloop restores CPython's signal handlers
-
Ctrl+Cis handled correctly, i.e.PyErr_SetInterruptis called when uvloop runs Python callbacks.
For the Windows port we don't need (1). We might need to play with Windows APIs to workaround (2), I suggest you to look at how CPython handles Ctrl+C on Windows.
As for socketpair–the whole subprocess handling code is complicated. I had to copy the logic from subprocess.py (and its C-level helper) to make uvloop's subprocess behave exactly like CPython's. Good news is that we can learn how subprocess.py uses Windows APIs, and copy that approach to uvloop.
Had a quick hack to make it buildable in windows (win7, python 3.5.2 x64): https://github.com/iceb0y/uvloop/commit/fa2b7db1d7940149bb8d16e856e7d1a642f3e6b8
build snippet: in /vendor/libuv, use python2
vcbuild.bat x64 release
in /, use python3, do the same as makefile
cython -3 -a -p uvloop/loop.pyx
python setup.py build_ext --inplace
Had a test with aiohttp server - the current stucking problem is that uvloop uses os_dup in multiple places to duplicate UVSocketHandle._fileno, where os_dup = os.dup and it duplicates posix file handles. In windows, posix file handle is implemented as an additional layer on NT handles where the handle numbers are not equal. I workaround this problem and aiohttp server works.
Notice that os_dup is also used to duplicate posix handle in uvloop/loop.pyx and uvloop/handles/process.pyx. So we need to differentiate between this two kinds of dups. Also note that duplicating socket handles in windows is fragile (see http://bugs.python.org/issue14522). If we can share socket by sharing handle objects instead of duplicating OS handles, we can resolve this problem and avoid the dup system call.
dup is introduced in 6e9c43bdecfad711c38c423fbada8fd8b86ddf7f
maybe we need to find another way to solve the problem described by the commit?
Your PR looks very good, thanks a lot for working on this. I believe we don't need to do anything special for signals or forks. As for dup, I'll fix a bug in asyncio and change how uvloop implements get_extra_info in a couple of weeks.
Sharing handle objects seems not working. uv still complains Assertion failed: (loop->watchers[w->fd] == w), function uv__io_stop, file src/unix/core.c, line 888.
Changing socket.socket to socket.fromfd, which uses _socket.dup which has special implementation for windows, without dup just works (tested in osx and windows). But I still feel a little strange to duplicate a socket handle.
What do you plan for dup?
Some workarounds to the signal based on @iceb0y 's commits: https://github.com/brickgao/uvloop/commit/eb9b9b7881fb1950e61033c3ba3934daf0a0c7e0
What do you plan for dup?
Just to remove it completely. Right now it's there to save uvloop from segfaulting when someone uses sockets from transports with add_writer API.
@1st1 I've fixed sendfile support on aiohttp master. Funny fact: on travis dupped socket was in blocking mode. Honestly I've not dug into but had added just socket.setblocking(False) call.
@1st1 I've fixed sendfile support on aiohttp master.
Super cool! Thanks!
Funny fact: on travis dupped socket was in blocking mode.
Wow. Maybe this is something worth investigating? Is it something that asyncio causes?
Some workarounds to the signal based on @iceb0y 's commits: brickgao@eb9b9b7
@1st1 I think the windows signal code is actually platform-independent
- Is it correct?
- Should we just use it for all platforms for simplification?
Funny fact: on travis dupped socket was in blocking mode.
Wow. Maybe this is something worth investigating? Is it something that asyncio causes?
@1st1 I cannot reproduce the problem locally. The only error what I have is travis report: https://travis-ci.org/KeepSafe/aiohttp/builds/157838616 Something like this:
resp = yield from sender().send(request, filepath)
File "/opt/python/3.5.0/lib/python3.5/asyncio/coroutines.py", line 105, in __next__
return self.gen.send(None)
File "/home/travis/build/KeepSafe/aiohttp/aiohttp/file_sender.py", line 148, in send
yield from self._sendfile(request, resp, f, file_size)
File "/opt/python/3.5.0/lib/python3.5/asyncio/coroutines.py", line 105, in __next__
return self.gen.send(None)
File "/home/travis/build/KeepSafe/aiohttp/aiohttp/file_sender.py", line 86, in _sendfile_system
''.join(headers).encode('utf-8'))
File "uvloop/loop.pyx", line 1831, in sock_sendall (uvloop/loop.c:29654)
ValueError: the socket must be non-blocking
sock.setblocking(False) has solved the problem.
I have no Idea why it works with some configurations but fails with others.
@iceb0y In the latest version of uvloop I've redesigned how signals are implemented. No more libuv or system API calls, I now only use Python's signal module. This should help with the Windows port.
tested on windows - the signal handling code doesn't work on windows, probably due to usage of socketpair.
Loop._setup_signals -> Loop._add_reader -> UVPoll.new -> UVPoll._init -> uv.uv_poll_init -> OSError: [Errno 4050] Unknown error.
Why does signal handling use a socketpair which seems to post notification to the loop?
FWIW I'm working on to add Windows support in https://github.com/MagicStack/uvloop/tree/win. Any help is welcome.
For those on windows 10, I believe you can use bash on ubuntu and install uvloop on there for now, or at least that seems to have worked for me.
For those on windows 10, I believe you can use bash on ubuntu and install uvloop on there for now, or at least that seems to have worked for me.
Great! FWIW I don't have time to focus on the IOCP win port right now, but I can definitely help with reviews/suggestions if someone is interested in it.
Is it support "Bash on Ubuntu on Windows"?
It's running on Windows Subsystem for Linux（WSL）。
A side note on 2017/02/16 for those who are interested later.
I come across these issues today (actually I want to install sanic), and try the latest windows branch commit https://github.com/MagicStack/uvloop/commit/3c741b4c27fec94569ff8c65222e4f314cf44079.
I git clone the repo and try to python setup.py install. I meet an error complaining python 2.7 not found because of https://github.com/MagicStack/uvloop/blob/win/setup.py#L227. (It's ironical because uvloop is designed for Python 3.5 +.) Python 2.7 is required for gyp, so I installed Python 2.7 and manually change that line to hard code my Python 2.7 path.
After that, I could successfully build and import uvloop! And the hello world of sanic works, too.
I also run the tests/. These tests failed:
test_dns.py
test_sockets.py
test_tcp.py
test_udp.py
Bash on Windows is Windows Subsystem for Linux. You are not running a windows executable, you are quite literally running (at the time of your post) Ubuntu 14.04 x86-64. WSL is not a production tool, it exists to let windows based developers write binaries that will be deployed on real ubuntu 14.04 (and now 16.04) machines.
WSL runs real linux binaries on the windows kernel ... unreliably. It's not a real solution to running on windows.
