Various issues when `File` is used with UNIX special files

[oblique]

The problem

I was working on a toy project and I couldn't get /dev/ptmx to work properly. To my surprise I found that File is actually using spawn_blocking instead of mio or epoll. IMO this is costly, but I guess you balanced this cost with the cache buffer.

Event if this implementation works well with simple files, it doesn't work at all with some special files because each special file has its own behavior.

Personally I worked only with two:

1. /dev/net/tun
   • This special file it is used to create a network tunnel.
   • Each read() syscall always return one network packet, if buf size if smaler than the packet data length, then buf is filled with truncated data, the rest data are just lost.
   • Each write() syscall must write one complete network packet.
2. /dev/ptmx
   • This special file is used to create a master PTY.
   • A read() syscall reads the stdout of the slave PTY, which means: no output for 5 minutes = blocks for 5 minutes.
   • Each write() syscall writes to stdin of the slave PTY and must be immediate (i.e. not cached).

In both cases cache can create misbehavior and in both cases spawn_blocking is very costly.

Proposal

Under UNIX systems mio must be used. No cache buffering must be applied since it can create misbehavior. With this we have one more benefit, File::from_raw_fd can be used to as an equivalent of mio's EventedFd.

If you really want cache then we can perform once the fstat() syscall on file descriptor and based on stat.st_rdev we can choose the threaded/buffered implementation or the evented/unbuffered implementation.

cc @yoshuawuyts @stjepang

[oblique]

This is also related to #594.

[lparam]

@oblique Have you solved this problem?

[oblique]

In general I was wrong about the File being able to be used with mio, because regular files do not support asynchronous IO by the OS (unless if io-uring is used). So File::from_raw_fd will be a wrong approach.

For the special files I now use async_io::Async<File> from async-io crate. Just keep in mind this can not be used with regular files, otherwise executor may block.