Microsoft Windows not fully supported

[kaimac1]

C:\home\aiocoap (master)
λ py -3 server.py
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<create_server_context() done, defined at C:\home\aiocoap\aiocoap\protocol.py:520> exception=OSError(10022, 'An invalid argument was supplied', None, 10022, None)>
Traceback (most recent call last):
  File "C:\Python34\lib\asyncio\tasks.py", line 238, in _step
    result = next(coro)
  File "C:\home\aiocoap\aiocoap\protocol.py", line 535, in create_server_context
    transport, protocol = yield from loop.create_datagram_endpoint(protofact, family=socket.AF_INET6)
  File "C:\Python34\lib\asyncio\base_events.py", line 744, in create_datagram_endpoint
    waiter)
  File "C:\Python34\lib\asyncio\selector_events.py", line 90, in _make_datagram_transport
    address, waiter, extra)
  File "C:\Python34\lib\asyncio\selector_events.py", line 983, in __init__
    super().__init__(loop, sock, protocol, extra)
  File "C:\Python34\lib\asyncio\selector_events.py", line 513, in __init__
    self._extra['sockname'] = sock.getsockname()
OSError: [WinError 10022] An invalid argument was supplied

and

  C:\home\aiocoap (master)
λ py -3 clientGET.py
Traceback (most recent call last):
  File "clientGET.py", line 34, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "C:\Python34\lib\asyncio\base_events.py", line 316, in run_until_complete
    return future.result()
  File "C:\Python34\lib\asyncio\futures.py", line 275, in result
    raise self._exception
  File "C:\Python34\lib\asyncio\tasks.py", line 238, in _step
    result = next(coro)
  File "clientGET.py", line 20, in main
    protocol = yield from Context.create_client_context()
  File "C:\home\aiocoap\aiocoap\protocol.py", line 510, in create_client_context
    transport, protocol = yield from loop.create_datagram_endpoint(protofact, family=socket.AF_INET6)
  File "C:\Python34\lib\asyncio\base_events.py", line 744, in create_datagram_endpoint
    waiter)
  File "C:\Python34\lib\asyncio\selector_events.py", line 90, in _make_datagram_transport
    address, waiter, extra)
  File "C:\Python34\lib\asyncio\selector_events.py", line 983, in __init__
    super().__init__(loop, sock, protocol, extra)
  File "C:\Python34\lib\asyncio\selector_events.py", line 513, in __init__
    self._extra['sockname'] = sock.getsockname()
OSError: [WinError 10022] An invalid argument was supplied

[kaimac1]

If I modify the create_datagram_endpoint call in create_server_context to this, and remove the setsockopt and bind calls:

transport, protocol = yield from loop.create_datagram_endpoint(protofact, local_addr=('127.0.0.1', 9999))
#transport._sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
#transport._sock.bind(bind)

And then modify the call in create_client_context:

transport, protocol = yield from loop.create_datagram_endpoint(protofact, remote_addr=('127.0.0.1', 9999))

Then if I request coap://localhost:9999/time with clientGET.py, it is sort of working:

λ py -3 clientGET.py
Result: 2.05 Content
b'2015-08-24 12:17'
Exception ignored in:

Not sure yet what the root cause is. Somebody else had the same problem here: http://stackoverflow.com/questions/31371055/trying-to-connect-to-coap-resource-with-python-library

[chrysn]

mh, the options in create_datagram_endpoint you've changed basically turn everything into ipv4-only.

while windows is not exactly on my planned list of supported platforms, it'd be nice if it worked there too; suggestions on how to properly open an ipv4 and ipv6 socket on windows are welcome.

(note that it might not be possible to open a IPv6_V6ONLY=0 socket on windows at all, even some BSDs refuse that, and i might need to adapt the endpoint implementation to support multiple interfaces for them too).

[koikos]

Hi, if you see issue: https://bugs.python.org/issue6926 then you my find that constant IPPROTO_IPV6 is not included in socket module due to some compilation issues. The workaround is simply to declare (according to RFC 3542 section 2.1.1) IPPROTO_IPV6 = 41 and use it instead of socket.IPPROTO_IPV6.

[conradek]

koikos, can you please provide an example of how this declaration is made?

[chrysn]

With the latest HEAD, most socket stuff is packed in aiocoap/transports/udp6.py. If you explicitly spell out the IPPROTO_IPV6 when undefined, is that sufficient to make aiocoap usable on Windows?

[ChangSam]

HI chrysn, have this issue been sloved yet ? And what's your " explicitly spell out the IPPROTO_IPV6 " means ??

[chrysn]

There have not been any updates; I don't have any version of Windows in use, so I need to rely on user feedback and patches.

"spell out the IPPROTO_IPV6" refers to koikos' comment about using the constant value instead of the import. Mapped to the current code base, that would mean replacing all occurrences of socket.IPPROTO_IPV6 with 41. In the mean time, as reported by @itsweet just some days ago (https://github.com/chrysn/aiocoap/issues/37), this affects the IN module as well; as a first workaround, you could probably just drop all lines that use that module. (Currently, that would mean dropping the import IN line, dropping the sock.setsockopt(..., IN.IP_RECVERR...) lines, and replacing the complete if cmsg_type == IN.IPV6_RECVERR parts with only the else clause's self.log.info line, all in aiocoap/transports/udp6.py. This should be the most basic thing to test in order to find out whether anything further hinders Windows compatibility.

Of course, if those constants were defined somehow in Windows as well, instead of dropping code, the proper constants for Windows could be substituted like should be for IPPROTO_IPV6. On my system, IN.IPV6_RECVERR is 25, and IN.IPV6_PKTINFO is 50 -- please test doing that as well.

[ChangSam]

hello chrysn !

1. which file or where you "" replacing all occurrences of socket.IPPROTO_IPV6 with 41 ""? Is this your means ? sock.setsockopt(41, socket.IPV6_V6ONLY, 0) (replace it directly??) or not ?
2. When I run my python file, it occurred "NO module name IN" so, I need to drop all contents that is about " IN " module as you write in the second paragraph instead of finding IN.IPV6_RECVERR or IN.IPV6_PKTINFO in windows system, right ?
3. I would like to know that have you ever run your python file successfully after you replace these values of contents ?

Thank you !!!!

[chrysn]

On Sun, Jun 12, 2016 at 12:47:58AM -0700, ChangSam wrote:

1. which file or where you "" replacing all occurrences of socket.IPPROTO_IPV6 with 41 ""? Is this your means ? sock.setsockopt(41, socket.IPV6_V6ONLY, 0) (replace it directly??) or not ?

The only file this should need replacement in is aiocoap/transports/udp6.py. Yes, sock.setsockopt(41, ...) is what it could look like for testing.

1. When I run my python file, it occurred "NO module name IN" so, I need to drop all contents that is about " IN " module as you write in the second paragraph instead of finding IN.IPV6_RECVERR or IN.IPV6_PKTINFO in windows system, right ?

Well, finding IN.IPV6_RECVERR is one step in dropping all contants about the IN module; I think I outlined all occurrences in the "Currently" sentence, though, so you just need to drop what I mentioned there.

1. I would like to know that have you ever run your python file successfully after you replace these values of contents ?

The test suite passes after replacing all occurrences of socket.IPPROTO_IPV6 with 41. Without the IN module, two tests fail -- but that's expected, because some functionality is not available immediatly (instead of getting a "Host unreachable", only timeouts show up -- not absolutely essential, but the test suite will fail).

You can limit the tests to the server-side tests (python3 -m unittest tests.server), those should pass even without the IN module.

In case this works with github mail replies, I've attached the simplified udp6.py file; it contains precisely the the previous comment.

[chrysn]

@ChangSam, could you try again with current master and an up-to-date Python? The IN module failures now have a fallback, and with https://bugs.python.org/issue6926 closed, the numbers in socket should be complete again.

[ChangSam]

Hello chrysn: Thanks for your reply ! but it seems still have some issues in windows environments (0001.jpg) ( I use IPv6 as my Coap url ) and if there are other fix solution will be great!! By the way, This project running very well in the linux environments, Thanks a lot !!!!!

2016-11-17 16:20 GMT+08:00 chrysn [email protected]:

@ChangSam https://github.com/ChangSam, could you try again with current master and an up-to-date Python? The IN module failures now have a fallback, and with https://bugs.python.org/issue6926 closed, the numbers in socket should be complete again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chrysn/aiocoap/issues/27#issuecomment-261183458, or mute the thread https://github.com/notifications/unsubscribe-auth/AQJUysh8oz0i6vnUUYDohiqIKoVA2rwtks5q_A5KgaJpZM4FtfbZ .

[chrysn]

@ChangSam, the attachment got stripped, but i got myself guest access to a Windows machine now. @2trc, could you subscribe here too? then i can close #52 and focus windows efforts here.

The sub-issues seem to be:

• Asyncio's sock.getsockname() seems to be untimely or otherwise wrong (that's the WinError 10022 also discussed on stackoverflow), but this looks like a Python problem to me. i've crudely removed the line where the exception is raised from from the system library; obviously, that's something one can only do during development if at all.
• aiocoap-client's readline import got a try-except-pass guard around it, given that readline is completely optional there, and just not available on windows.
• socket.IPPROTO_IPV6 is not defined on windows; that can be fudged by just setting IPPROTO_IPV6 to a constant value (41) – should go into aiocoap/util/socknumbers then
• setsockopt IPV6_RECVPKTINFO does not work, because RECVPKTINFO is not defined. even if we defined that, it would be no good, because
• (and that's the gravest) there is no recvmsg on windows. recvmsg is used instead of recvfrom to get information about the package's destination address (that bites when a host has multiple fully-routed ip addresses), and information about icmp errors (without, waiting for errors does become boring).

i see three ways to go on about this:

• a) implement a "stupid" udp transport that only uses recvfrom, ignores icmp, and just will just randomly work or not work on systems with multiple ip addresses
• b) implement a udp transport that only uses recvfrom, but internally creates a connect()ed socket for each communication partner for outgoing connections, and/or (?) a socket for each local interface (which also means enumerating ip addresses and binding to new ones as they appear to simulate being bound to "::" / "0.0.0.0"). that should allow receiving (at least some) errors and gives a sending address on returns.
• c) implement a udp transport that is windows specific; it could use any windows api that's suitable, as posix systems would just use the existing one.

c) looks like the sanest one – i just have no clue where to find the suitable api documentation, and whether that api would be available in python. b) sure would be a lot of work, and while i had contemplated doing it before i found recvmsg, it'd be too much effort for me just for windows support. a) sound like asking for bug reports.

[ChangSam]

Hello chrysn: I wanna ask a question about observe CoAP, and thanks for answer to me The IEFT defines that the observer can subscribe to listen for "any change" in the state of an observable subject. which means that if the "state change" or "after a certain second(max age time)doesn't appear state change" ,it will return the observe payload so ,if the observe server uses "time" as observable_resource ,by IEFT difinision, it will sent oberserve value return to observer "every second" because time update every second,rather than by a certain (max age time) to sent.

Or we can just say that because time varies to fast, so we only need to sent by certain seconds, is this statement also follow observe rules by IEFT? thanks!!! Sam Chang

2016-12-13 3:17 GMT+08:00 chrysn [email protected]:

@ChangSam https://github.com/ChangSam, the attachment got stripped, but i got myself guest access to a Windows machine now. @2trc https://github.com/2trc, could you subscribe here too? then i can close #52 https://github.com/chrysn/aiocoap/issues/52 and focus windows efforts here.

The sub-issues seem to be:

• Asyncio's sock.getsockname() seems to be untimely or otherwise wrong (that's the WinError 10022 also discussed on stackoverflow), but this looks like a Python problem to me. i've crudely removed the line where the exception is raised from from the system library; obviously, that's something one can only do during development if at all.
• aiocoap-client's readline import got a try-except-pass guard around it, given that readline is completely optional there, and just not available on windows.
• socket.IPPROTO_IPV6 is not defined on windows; that can be fudged by just setting IPPROTO_IPV6 to a constant value (41) – should go into aiocoap/util/socknumbers then
• setsockopt IPV6_RECVPKTINFO does not work, because RECVPKTINFO is not defined. even if we defined that, it would be no good, because
• (and that's the gravest) there is no recvmsg on windows. recvmsg is used instead of recvfrom to get information about the package's destination address (that bites when a host has multiple fully-routed ip addresses), and information about icmp errors (without, waiting for errors does become boring).

i see three ways to go on about this:

• a) implement a "stupid" udp transport that only uses recvfrom, ignores icmp, and just will just randomly work or not work on systems with multiple ip addresses
• b) implement a udp transport that only uses recvfrom, but internally creates a connect()ed socket for each communication partner for outgoing connections, and/or (?) a socket for each local interface (which also means enumerating ip addresses and binding to new ones as they appear to simulate being bound to "::" / "0.0.0.0"). that should allow receiving (at least some) errors and gives a sending address on returns.
• c) implement a udp transport that is windows specific; it could use any windows api that's suitable, as posix systems would just use the existing one.

c) looks like the sanest one – i just have no clue where to find the suitable api documentation, and whether that api would be available in python. b) sure would be a lot of work, and while i had contemplated doing it before i found recvmsg, it'd be too much effort for me just for windows support. a) sound like asking for bug reports.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chrysn/aiocoap/issues/27#issuecomment-266524031, or mute the thread https://github.com/notifications/unsubscribe-auth/AQJUyk5qG-XX27dbfrv-ksYNSuugVgm8ks5rHZ3CgaJpZM4FtfbZ .

[chrysn]

@tcareynokia has implemented a) in https://github.com/tcareynokia/aiocoap/commit/f1e8e5ad3d8233ee67d90350d3ec0078119301d4 . tim, could you tell about how things worked out with your changes?

[tcareynokia]

Hi - I have it "working" but there are problems - related to windows.

1. I needed to work around ancillary data with the comments suggested in previous posts
2. I needed to work around the problems with IPv6 endpoints by forcing it to IPv4 only
3. There was a problem in the timeout logic where you were deleting a request but the request wasn't in the dictionary - I caught the exception but I can't determine what to throw instead if anything - right now I pass. The code is simply a hack; certainly not ready for a pull request. I have windows 7 and windows 10 that I run this on if you want me to check anything out.

BR, Tim

[chrysn]

Not a full solution, but the simple6 backend in the simple6 branch allows at least client operation; by setting AIOCOAP_CLIENT_TRANSPORT=simple6 in the environment, aiocoap-client and other programs using client contexts only should be usable under Windows.

Could someone who has a Windows instance around give this a try? coap://coap.me/ is a usable test server, given that aiocoap can't provide a test server on the same host.

[chrysn]

The simple6 branch is now gone and merged into master; please test whether the latest development version can run AIOCOAP_CLIENT_TRANSPORT=simple6 ./aiocoap-client coap://coap.me/ successfully.

[jeeger]

I think this only works if you actually have V6 connectivity, which is not available here.

I've followed this bug into aicoap – the problem for me seems to be that aiocoap requests AI_V4MAPPED addresses, but IPV6_V6ONLY is set on sockets by default on Windows. There does not seem to be a way to switch V6ONLY off in asyncio.

I've reported a python bug at https://bugs.python.org/issue36208. Additionally, I have "implemented" a "simple4" transport by copying the "simple6" transport and replacing every mention of V6 with V4. See PR #141 for this.

[chrysn]

@jeeger, I don't have a Windows machine to test, but could you try out the simple6 transport and just remove the family and flags lines there? I haven't run all the relevant tests yet, but looking at your PR it occurred to me that we might not need to keep two separate versions around.

[jeeger]

Uh, of course. There are only 2 relevant changes in the simple4 transport (removal of those two lines), but the addition of a new transport was the easiest way to integrate it "cleanly" into aiocoap.

What would you suggest as a way to pass configuration arguments to the transport? Then we would just remove AF_INET6 and the AI_V4MAPPED argument if this argument is passed.

[chrysn]

My hypothesis is that we don't need configuration here -- I'd go for just removing the two lines from the simple6 transport (which'd then be renamed to simpleclient transport or so). I can test that on Linux (it works for the simple cases) but could need help testing that on Windows.

(The simple transports should not rely on any structure of the address anyway but are more "make do with what's everywhere" alternatives for platforms that don't support all the POSIX flags to do it right.)

[jeeger]

Ah, okay! I'll make it another pull request then!

[jeeger]

I've removed the flags, however, I'm unable to test it on a device with V6 connectivity.

[chrysn]

Thanks, merged; that looks good and should make working on Windows less painful.

Still not closing this whole issue as the discussion on https://github.com/python/cpython/pull/11784 (following some links from the issue you opened at bpo) indicates that full support on Windows can be possible.

[chrysn]

This issue practically depends on https://bugs.python.org/issue36217 now; please open a dedicated issue if something doesn't work on Windows.

[chrysn]

I've posted a long summary of the current status in the MacOS thread, which applies to Windows as well.