Open special files in another thread

[ysbaddaden]

We can't detect without significant overhead whether the file at path might block for a while before calling open.

While O_NONBLOCK has no effect on regular disk files, special file types are a different story. Open with O_NONBLOCK will fail with ENXIO for O_WRONLY (no connected reader) but it will always succeed for O_RDONLY (regardless of a connected writer or not), then any attempt to read will return EOF, leaving no means to wait until a writer connects.

I thus rely on the blocking arg: when nil the file might be a special file type, so I check it's type, when false I consider it's a special file. If the file is a fifo (named pipe) or a character device, I open it in another thread to not risk blocking the current thread (and thus other fibers) until a reader or writer is also connected.

Opening regular files is unaffected: we still open directly, while opening a special file will now avoid blocking the current thread.

NOTE: we need the preview_mt flag to safely re-enqueue the current fiber from the bare thread.

Depends on #15754 and #15767.

[ysbaddaden]

After talking with @straight-shoota this PR might be a bit rushed/shallow.

We should take execution contexts into account.

1. The isolated context doesn't need to start a thread: we can block the current fiber/thread;

2. MT context could indeed detach the scheduler as Go does (can be moved to another waiting thread, or reattached upon return), and I guess we could have a pool of available threads (shared between contexts); an issue is how many maximum threads do we want :sweat_smile:

3. ST context might be more challenging: we could also detach the thread if we're just wrapping a syscall, it might be acceptable... as long as we don't expect thread locals :thinking:

That would allow a more general and useful solution. For example we could wrap addrinfo in addition to open.

[ysbaddaden]

Obsoleted by #15871.