[LibOS] `ioctl(FIONCLEX)` with file/socket/pipe opened with `O_CLOEXEC` flag doesn't work as expected

[llly]

Description of the problem

Current ioctl(FIONCLEX) only modifies internal flags and doesn't not touch the actual handle in Pal. https://github.com/gramineproject/gramine/blob/f91a2c18370858701a33c00620b0c7954f6779bf/libos/src/sys/libos_ioctl.c#L60-L61

When open, socket and pipe2 syscalls with O_CLOEXEC or SOCK_CLOEXEC flag create a host file descriptor with FD_CLOEXEC flag, it doesn't have a chance to clear the flag and vice versa. In Gramine, after forked child execve, the fd is closed due to CLOEXEC.

Steps to reproduce

run sample

int main(int argc, char** argv) {
    struct sockaddr_in local_addr = { 0 };
    socklen_t len = sizeof( struct sockaddr_in );
    if( argc ==2) {
        int s =atoi( argv[1]);
        CHECK(getsockname(s, (struct sockaddr*)&local_addr, &len));
        exit(0);
    }
    int s = CHECK(socket(AF_INET, SOCK_CLOEXEC | SOCK_STREAM, 0));
    CHECK(ioctl(s, FIONCLEX));
    if( 0 == fork()) {
        CHECK(getsockname(s, (struct sockaddr*)&local_addr, &len));
        char fd[25] = {0};
        snprintf (fd, sizeof(fd), "%d",s);
        char* argvs[] = {argv[0], fd , NULL};
        CHECK(execve(argv[0], argvs, NULL));
        exit(0);
    }
}

Expected results

No error printed.

Actual results

error at getsockname(s, (struct sockaddr*)&local_addr, &len) (line 202): Bad file descriptor

Additional information

This is the root cause of https://github.com/gramineproject/gsc/issues/80, a real python app. When debugger is enabled, python code uses WERKZEUG_SERVER_FD env and execve python again to start a debugger in order to use the socket created in parent python.

[P1:T1:python3.8] trace: ---- socket(INET, SOCK_CLOEXEC|STREAM, 0) = 0x3
[P1:T1:python3.8] trace: ---- ioctl(3, FIONCLEX, 0) ...
[P1:T1:python3.8] trace: ---- return from ioctl(...) = 0x0
[P3:T3:python3.8] trace: ---- execve("/usr/bin/python3.8", [/usr/bin/python3.8,/scripts/helloworld.py,], [LD_LIBRARY_PATH=/usr/lib/python3.8/lib:/lib:/lib/x86_64-linux-gnu:/usr/lib:/usr//lib/x86_64-linux-gnu,WERKZEUG_SERVER_FD=3,]) 

Fix

I think ioctl(FIONCLEX) need to passthrough to host by fs_ops->ioctl of some builtin_fs. But Linux-SGX Pal doesn't have a proper ocall to handle it. We have similar ocalls ocall_fionread and ocall_fsetnonblock. Maybe we have to add new ocall like int ocall_fsetcloexec(int fd, int cloexec);

Gramine commit hash

42434c583b10dc9d33a32485b6c95695e4bb72cb

[boryspoplawski]

Current ioctl(FIONCLEX) only modifies internal flags and doesn't not touch the actual handle in Pal.

This is correct and intended.

When open, socket and pipe2 syscalls with O_CLOEXEC or SOCK_CLOEXEC flag create a host file descriptor with FD_CLOEXEC flag, it doesn't have a chance to clear the flag and vice versa.

Host CLOEXEC does not matter, we always set that anyway.

In Gramine, after forked child execve, the fd is closed due to CLOEXEC.

The real problem is that FD_CLOEXEC is set in handles map only on fd creation and is never updated. We should always consult handle->flags (and remove this info from handles map) or update it flags in the map correctly. Also this ioctl is missing locking completely.

I think ioctl(FIONCLEX) need to passthrough to host by fs_ops->ioctl of some builtin_fs. But Linux-SGX Pal doesn't have a proper ocall to handle it. We have similar ocalls ocall_fionread and ocall_fsetnonblock. Maybe we have to add new ocall like int ocall_fsetcloexec(int fd, int cloexec);

No, please see above.

[dimakuv]

Yes, agreed with @boryspoplawski

[llly]

The real problem is that FD_CLOEXEC is set in handles map only on fd creation and is never updated. We should always consult handle->flags (and remove this info from handles map) or update it flags in the map correctly.

I see. libos_fd_handle and libos_handle are different handles. Need to sync the flag in them.

[boryspoplawski]

Or rather not keep the flags duplicated. This needs some investigation

[dimakuv]

Just some references for history (when I was reviewing #842):

• The libos_fd_handle::flags is for file descriptors (FDs). Described in Gramine here: https://github.com/gramineproject/gramine/blob/cf43bbca4c48456378b34621a1a271c6b3bd8148/libos/include/libos_handle.h#L228
• The libos_handle::flags is for file descriptions (aka LibOS handles, there is only one file description per real object, but can be multiple file descriptors to the same file description). Described in Gramine here: https://github.com/gramineproject/gramine/blob/cf43bbca4c48456378b34621a1a271c6b3bd8148/libos/include/libos_handle.h#L207

For FIOCLEX and FIONCLEX ioctls, see e.g. this man page: https://www.mkssoftware.com/docs/man3/ioctl.3.asp

For file descriptor flags vs file description flags, see https://man7.org/linux/man-pages/man2/fcntl.2.html