jupyterhub_moss
jupyterhub_moss copied to clipboard
spawner.py fails to parse when multiple gpu types in same node
Where a slurm compute node has multiple gpu types, as in my example below, the parser fails.
sinfo -a --noheader -o '%R %F %c %C %G %m %l' returns: gpu-private 1/0/0/1 16 8/8/0/16 gpu:turing:1,gpu:ampere:3 257595 infinite
This fails in lines 131-134 of spawner.py which makes an assumption there is only 1 GPU type.
try:
gpus_gres = gpus.replace("(", ":").split(":")
gpus_total = gpus_gres[2]
gpu = ":".join(gpus_gres[0:2]) + ":{}"
gpus_total then becomes "1,gpu" and fails an integer check.
The easiest fix here is to split on commas first and then loop through the different types of GPU, but that will likely throw up more things to deal with further down the line.