Repeating calls to connect_write_pipe with sys.stdout as a stream lead to FileExistsError exception

[hh-h]

• uvloop version: 0.11.3, 0.12.2, 0.13.0rc1
• Python version: 3.6, 3.7
• Platform: linux, macos
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: yes

Hello, this example doesn't work with uvloop.

import asyncio
import sys
import uvloop

uvloop.install()
loop = asyncio.get_event_loop()

async def main():
    transport, protocol = await loop.connect_write_pipe(
        asyncio.Protocol, sys.stdout
    )
    print("okay")

    transport2, protocol2 = await loop.connect_write_pipe(
        asyncio.Protocol, sys.stdout
    )
    print("okay")

loop.run_until_complete(main())

okay
Traceback (most recent call last):
  File "uvl.py", line 20, in <module>
    loop.run_until_complete(main())
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "uvl.py", line 16, in main
    asyncio.Protocol, sys.stdout
  File "uvloop/loop.pyx", line 2703, in connect_write_pipe
  File "uvloop/loop.pyx", line 2698, in uvloop.loop.Loop.connect_write_pipe
  File "uvloop/handles/pipe.pyx", line 170, in uvloop.loop.WriteUnixTransport._open
  File "uvloop/handles/pipe.pyx", line 29, in uvloop.loop.__pipe_open
FileExistsError: [Errno 17] File exists

please, see discussion here https://github.com/B2W-BIT/aiologger/issues/42#issuecomment-457283504

[1st1]

Interesting. Why do you think this is a good idea to connect two transports to the same fd?

[madkote]

@1st1 in case you log to stdout from different loggers, where both use stdout as stream handler.

[1st1]

Sure, but then you might have a race condition between these two protocols/transports not synchronizing their writes/reads to the FD. And if you want to do something like that anyways, the preferred solution would be probably to just os.dup() the FD for another protocol/transport pair, right?

[madkote]

@1st1 Thx. I do fully agree regarding os.dup() - it can be even a good workaround for the library mentioned above.

But there might be other scenarios (which I am not aware right now), where asyncio would work, and uvoop would not. So the question is - how far uvloop want to be similar to asyncio in usage.

[WouldYouKindly]

Just my two cents: we had cases when uvloop would raise FileNotFoundError, while the file was indeed there; also, sometimes writes would go to the wrong files. We did not manage to reliably reproduce it, and just ditched uvloop; asyncio works fine. It looked like uvloop mistook one file descriptor for another or something...

Non sure if it's relevant or not, and unfortunately cannot provide more info.

[1st1]

Yeah, this definitely needs to be investigated. If anyone wants to help I'd really appreciate that. I myself have no idea when I have time to debug this.

[1st1]

@hh-h The reason the original example does not work is because libuv requires every FD to be registered once. So in order to fix this in uvloop, uvloop itself will have to dup FDs behind the scenes. Which is a bit suboptimal, but I'll think about it.

@WouldYouKindly

Just my two cents: we had cases when uvloop would raise FileNotFoundError, while the file was indeed there; also, sometimes writes would go to the wrong files.

The only way I can imagine this happening is if you somehow close FDs without closing the transports/protocols that wrap them. In this case, uvloop might have old wrong transports wrapping "new" FDs. Or something else that's crazy like this.

What it really means is that you program might work in asyncio and not in uvloop now, but it's only a matter of time until it starts to break horribly in asyncio too. I might be wrong here, but I suggest you to better investigate what's really happening on your end. uvloop is used in production in enormous deployments and this is the first time I hear about "writing to wrong files". So if I were you I'd check just in case. Obviously I'd love to say something more definitive than just speculating, but I need a piece of code to reproduce the bug you describe.

[guishake]

Hello, I confirm that this error appears using uvloop and disappears removing this lib, which is a pity because uvloop is really really performing compared to standard asyncio, is there any plan to fix the issue ?