loti-examples icon indicating copy to clipboard operation
loti-examples copied to clipboard

`cqes->res` value is not updated after io_uring_wait_cqe call.

Open tofes opened this issue 3 years ago • 1 comments

The issue was discovered when running the "webserver_liburing" example. The values of cqe was printed out after the following code was called the first time:

        int ret = io_uring_wait_cqe(&ring, &cqe);
        if (ret < 0)
            fatal_error("io_uring_wait_cqe");

        struct request *req = (struct request *) cqe->user_data;
        if (cqe->res < 0) {  // <------ ALWAYS TRUE
            fprintf(stderr, "Async request failed: %s for event: %d\n",
                    strerror(-cqe->res), req->event_type);
            exit(1);
        }

Unexpectedly cqe->res < 0 was always true. The value of cqe->res was -5, input/output error. The sqe->opcode was 13 (IORING_OP_ACCEPT). Using ss command helped to show that the socket was in LISTEN state, which corresponded with the ACCEPT since no socket connection was established. LISTEN 0 10 0.0.0.0:8000 0.0.0.0:* users:(("webserver_liburing",pid=58904,fd=3))

The issue is reproduced on the following system environment:

 Operating System: Ubuntu 20.04.3 LTS
           Kernel: Linux 5.5.0-050500-lowlatency
     Architecture: x86-64

Why was the cqe->res not updated by kernel? Printing the errno shows different value.

tofes avatar Feb 03 '22 03:02 tofes

Suggest to update the server_loop function as follows:

void server_loop(int server_socket) {
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);

    add_accept_request(server_socket, &client_addr, &client_addr_len);

    while (1) {
        struct io_uring_cqe *cqe;                // <---Enable printing out a different cqe for each loop.
        int ret = io_uring_wait_cqe(&ring, &cqe);
        printCQ(&ring.cq,0);
        if (ret < 0)
            fatal_error("io_uring_wait_cqe");
        struct request *req = (struct request *) cqe->user_data;
        if (cqe->res < 0) {
            fprintf(stderr, "Async request failed: %s for event: %d\n",
                    strerror(-cqe->res), req->event_type);
            io_uring_cqe_seen(&ring, cqe);  // <--- Added to mark the failed operation to be processed.
            exit(1);
        }

        switch (req->event_type) {
            case EVENT_TYPE_ACCEPT:
                add_accept_request(server_socket, &client_addr, &client_addr_len);
                add_read_request(cqe->res);
                free(req);
                break;
            case EVENT_TYPE_READ:
                if (!cqe->res) {
                    fprintf(stderr, "Empty request!\n");
                    break;
                }
                handle_client_request(req);
                free(req->iov[0].iov_base);
                free(req);
                break;
            case EVENT_TYPE_WRITE:
                for (int i = 0; i < req->iovec_count; i++) {
                    free(req->iov[i].iov_base);
                }
                close(req->client_socket);
                free(req);
                break;
        }
        /* Mark this request as processed */
        io_uring_cqe_seen(&ring, cqe);
        
    }
}

tofes avatar Feb 03 '22 23:02 tofes