NetworkBlockDevice
Performance optimization
How does nbd manage processes and threads? I am looking into this to find out what optimizations may be required and what settings to watch for as I start to use nbd-server more. Is there anything I need to watch out for when it comes to performance?
Is there a limit to the number of processes that are started? Is there a way to manage the number of processes?
The man page describes that the nbd-server will read requests in a main thread and handle those requests in separate threads. It appears, in my test environment, that a new process is always started when needed, as a client connects.
What is the correct way to get the number of threads?
When I set max_threads to 30, threads always stays at 1. I am checking this with cat /prod/<nbd process id>/status | grep Thread. Does the fact that I am making read-only requests to the server have something to do with this?
My environment
(for context)
- PXE bootable
- ~1000-2000 PXE booted read-only connections (to nbd server)
- raw file image created from a Cloud-init install, Ubuntu 22.04
- overlayfs to create a read only operating system (changes are not save after a reboot)
- nbd-server
- 1 nbd conf.d export (soon to be 5+)
Thank you in advance for your help.
L
nbd-server uses glib's GThreadPool to implement async handling of requests. That has a function to set the maximum number of threads, which is what we use when you set max_threads.
At this point, nbd-server still uses fork-per-child semantics, and the threadpool handling is only started after the fork. While I tried improving that situation a few times in the past, unfortunately that resulted in security issues every time, so I'm not going to do that again without a major refactoring of nbd-server first. I still want to do that at some point in the future, but unfortunately I don't have much time to do this right now.
However, the issue still exists, and I do still want to keep it on my radar, so reopening.