prmon icon indicating copy to clipboard operation
prmon copied to clipboard

Improve nvidia-smi output parsing

Open amete opened this issue 1 month ago • 0 comments

Depending on the hardware/driver combination, nvidia-smi pmon can report partial utilization data, for example, fb memory usage may be available, while sm and mem are reported as -:

[2025-11-11 08:22:33.264] [nvidiamon] [debug] nvidiamon::update_stats got the following output (4): 
0 -> # gpu         pid   type     sm    mem    enc    dec    jpg    ofa     fb   ccpm    command 
1 -> # Idx           #    C/G      %      %      %      %      %      %     MB     MB    name 
2 ->     0    2433046     C      -      -      -      -      -      -  19044      0    python         
3 ->     0    2448507     C      -      -      -      -      -      -   7602      0    python3

When parsing this output, the stream fails at the first - because it’s non-numeric (we expect sm and mem to be numeric), and subsequent fields like fb (which are valid) are lost:

[2025-11-11 08:22:33.264] [nvidiamon] [debug] Bad read of line:     0    2433046     C      -      -      -      -      -      -  19044      0    python         
Parsed to: 0 2433046 C 0 0     0  
StringStream status: good()=0 eof()=0 fail()=1 bad()=0

[2025-11-11 08:22:33.264] [nvidiamon] [debug] Bad read of line:     0    2448507     C      -      -      -      -      -      -   7602      0    python3        
Parsed to: 0 2448507 C 0 0     0  
StringStream status: good()=0 eof()=0 fail()=1 bad()=0

[2025-11-11 08:22:33.264] [nvidiamon] [debug] Parsed: gpufbmem: 0;gpumempct: 0;gpusmpct: 0;ngpus: 0;

To deal with this properly, parsing should not abort when encountering -. Fields like fb should still be read correctly even if other metrics are missing.

This can be achieved by reading all fields as strings first, and convert them to integers afterward, handling the special - case (e.g., treat it as 0). This would allow preserving valid numeric fields like fb while gracefully handling unavailable metrics.

amete avatar Nov 11 '25 21:11 amete