gpustat
gpustat copied to clipboard
wookayin
use too much CPU resource
Hi, gpustat -i uses about 80% CPU on my machine, is it expected or a bug?
In contrast, nvidia-smi -l 1 uses less than 10%.
OS: Ubuntu 18.04 Nv Driver: 410.48 CUDA: 10.0.130 CPU: AMD Threadripper 1900x GPU: 2080Ti + 1080
Can you test if watch -n 1 nvidia-smi and watch -n gpustat use the same amount of CPU time.
I tested, and with watch -n 1 they use the same amount of CPU time (about 0-20%).
Hi, I just found that the CPU time problem of gpustat -i can be solved by running nvidia-smi daemon first.
A difference is that in the watch mode (i.e. gpustat -i) handle resources are fetched at every time step, which is somewhat expensive. Therefore we could optimize in a way that GPU handles are fetched only once in the beginning, and use the (cached) resources. This would be possible in the watch mode as the gpustat process won't terminate until interrupted.
Top four most expensive operations:
36.50% 36.50% 4.59s 4.59s nvmlDeviceGetHandleByIndex (pynvml.py:945)
16.50% 16.50% 1.85s 1.85s nvmlDeviceGetPowerUsage (pynvml.py:1289)
9.50% 9.50% 1.18s 1.18s nvmlDeviceGetUtilizationRates (pynvml.py:1379)
7.50% 7.50% 0.805s 0.805s nvmlDeviceGetComputeRunningProcesses (pynvml.py:1435)
Working on this as #61.
In my case querying power usage is most expensive, so I made it optional whenever possible. Could anybody check whether it leads to less CPU usage?
A difference is that in the watch mode (i.e.
gpustat -i) handle resources are fetched at every time step, which is somewhat expensive. Therefore we could optimize in a way that GPU handles are fetched only once in the beginning, and use the (cached) resources. This would be possible in the watch mode as the gpustat process won't terminate until interrupted.
But I still have no idea of the difference between "watch -n 1 gpustat" and "gpu -i 1". Both of them need to call print_gpustat() every tick, while 'watch' requires additional step to parse command line arguments again and again. So intuitively the former should took longer.
BTW, here is the source code of 'watch' if needed: watch.c, where I found nothing useful : (
Ter,
The most time is not spent on parsing CLI options actually, which only requires several us at most.
On Mar 18, 2021, at 10:42 PM, Ter @.***> wrote:
A difference is that in the watch mode (i.e. gpustat -i) handle resources are fetched https://github.com/wookayin/gpustat/blob/v0.5.0/gpustat/core.py#L372 at every time step, which is somewhat expensive. Therefore we could optimize in a way that GPU handles are fetched only once in the beginning, and use the (cached) resources. This would be possible in the watch mode as the gpustat process won't terminate until interrupted.
But I still have no idea of the difference between "watch -n 1 gpustat" and "gpu -i 1". Both of them need to call print_gpustat() every tick, while 'watch' requires additional step to parse command line arguments again and again. So intuitively the former should took longer.
BTW, here is the source code of 'watch' if needed: watch.c https://gitlab.com/procps-ng/procps/-/blob/master/watch.c, where I found nothing useful : (
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wookayin/gpustat/issues/54#issuecomment-801984777, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCYKDBSTNB2M3VNKCKC2RLTEIGMNANCNFSM4GCTJF2Q.
Has this issue been resolved? I am observing this behavior from https://github.com/ray-project/ray/ when we run gpustat.new_query() repetitively at GCE.
In the recent versions of pynvml, nvmlDeviceGetHandleByIndex doesn't seem to be a bottleneck according to profiling result (If this is still slow, please let me know) so I did not optimize on redundant calls of nvmlDeviceGetHandleByIndex. #166 makes nvmlInit() called only once, so it should have some performance benefit.