improper GPU utilization

[ibrahimrazu]

Thanks a lot for sharing your code. While training the NYU dataset, seems like it is not utilizing GPU properly and training time is significantly higher. Could you please tell me how did you configure the GPU there? I tried to configure the GPU as following: It is noteworthy that i've checked the GPU are working, but utilization is pretty low, almost minimal

[dumyy]

You are using cpu to train the model, so it must be very slow! I think there is somehing wrong about your gpu config. You shoud make sure that you can use tensorflow-gpu with right gput config ( like cuda cudnn)

[dumyy]

316MB is the memory that gpu is free.

[ibrahimrazu]

Thanks a lot for your advice. Yes, its working fine now. I have installed tensorflow-gpu at my conda environment. However, only one GPU is being utilized among all. I tried several combinations, but failed to assign the task to multi-gpus

[dumyy]

Sorry, I have never used multi-gpu config to train the model, so maybe that I can't help you directly.

[ibrahimrazu]

No problem, thanks a lot! i'll post here if i can do that

[eparvizi]

@ibrahimrazu did you figure out how to enable multi-gpu processing? @dumyy my GPU utilization is low, ranging between 15-30%. How would you parallelize and separate CPU/GPU tasks to maximize GPU utilization?

[ibrahimrazu]

@eparvizi not yet unfortunately. however my single GPU utilization is always highest. I tried with CUDA_VISIBLE_DEVICES=0,1,2,3 but it always uses the first GPU you mention.

[eparvizi]

@ibrahimrazu have you tried this? https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus

[ibrahimrazu]

Still not. But i’ll try. Thanks!

[mslqing]

@ibrahimrazu Hello, how does your program run on GPU? I did not find the program code in the following image in the source program.