FR: let userspace provide the worker threads

[dcolascione]

I've never liked this design pattern in which the kernel creates worker threads automatically on behalf of userspace: userspace has a hard time controlling the number of threads and their characteristics, e.g., affinity, priority, and name. Right now, io_uring does just that: the async backend creates work queues and kernel threads to process the work. There's only limited control, all done in an ad-hoc way, over what these threads look like and how many of them there are. (For the program I've been writing, I see one wq worker pegged at 100% of one core and the system reluctant to make more.)

I wish I could bring my own worker threads instead. Why not let users create their own threads and let them run io_uring work through a new io_uring_enter command? A user-provided worker thread would stay blocked in the kernel for a long time, continually doing async io_uring work that's been submitted, then go back to userland upon receipt of a signal, expiration of a timeout, or some maximum number of operations being done.

This approach would be exactly as efficient as io_uring automatically making its own workers (because the user provided worker would stay in the kernel while it does many operations and wouldn't hop in and out of the kernel for each piece of work it does) but would give userspace greater control over how these workers look without having to introduce a brand new io_uring API for every aspect of the wq worker thread that a user might want to control.

If you combine this new io_uring_enter flag and combine it with a new "never automatically create worker threads -- I'll bring my own" flag to io_uring setup, you give the user total control over how work is done.

[axboe]

I'm actually working towards something that would, as a side effect, probably also make this possible. See this branch:

https://git.kernel.dk/cgit/linux-block/log/?h=io_uring-worker.v2

The idea is to have the workers be cloned from the original task, instead of having the workers be kernel threads that attempt to grab whatever context they need. If this can get pulled off, then it would not be a stretch to allow a way to register regular userspace threads with io_uring and use those instead of cloning new ones when needed.

[HowHsu]

Hi @dcolascione , I'm working on something like this, hopefully it can come out before long.

[axboe]

@HowHsu that'd be great! Should indeed be feasible now that we have real thread workers.