volatile GPU-util is pretty low

[lmy931]

Hi, I run the command "nvidia-smi" and I found the "volatile GPU-util" is around 20%. Do you think it has not fulfilled my GPU potential? The GPU memory-usage seems to be good. It shows 7865MiB / 7951MiB.

Also, "volatile GPU-util" is not always 20%, it can be 20% for one second and drop to 0% in the next second, and then go back to 20% again. I mean the figure seems to be 0%-20%-0%-20%... I can hear my GPU fan run fast-slowly-fast-slowly...

[edvardHua]

In GTX 1080Ti, the "volatile GPU-util" is 0%~80%.

If you model already fixed, you can delete the validate code in training to reduce circumstance of 0% "volatile GPU-util".

[lmy931]

Thanks. I will try. I set the params['pred_image_on_tensorboard'] to be false. I think the validation should be stopped.

[leviethung2103]

@edvardHua

Hi edvardHua, I've tried to start the training as the ordinary way, However, I met the same problem with GPU usage. I got only 50~63% of GPU. This is an example of log during training.

step 100, loss = 6285.83, last_heat_loss = 2424.42 (72.6 examples/sec; 0.220 sec/batch)

I've tried to delete the validate code but the training is extremely slow.

How can I utilize the full potential of GPU with your training code? Thank you in advance.