AnyDoor icon indicating copy to clipboard operation
AnyDoor copied to clipboard

It is failed when I try to train on Tesla V100GPU, what is your GPU type?

Open EricShow opened this issue 1 year ago • 3 comments

EricShow avatar Dec 19 '23 02:12 EricShow

I use A100-80G to train the model. For small memories, you could find out whether there exist more tricks for Pytorch Lightening. Or modify run_train_anydoor.py to set batch size = 1 and use batch accumulation tech.

XavierCHEN34 avatar Dec 19 '23 02:12 XavierCHEN34

I use A100-80G to train the model. For small memories, you could find out whether there exist more tricks for Pytorch Lightening. Or modify run_train_anydoor.py to set batch size = 1 and use batch accumulation tech.

Thank you,my batchsize already set 1,I see that your batchsize sets 16,so the 80G of memory can accommodate batch size 16?My V100 is 32G memory

EricShow avatar Dec 19 '23 07:12 EricShow

I use A100-80G to train the model. For small memories, you could find out whether there exist more tricks for Pytorch Lightening. Or modify run_train_anydoor.py to set batch size = 1 and use batch accumulation tech.

Thank you,my batchsize already set 1,I see that your batchsize sets 16,so the 80G of memory can accommodate batch size 16?My V100 is 32G memory

my model failed when i trained with 2 tesla V100 with batch size one.

zz8615 avatar Jan 24 '24 16:01 zz8615