Memory usage is large before training, and while training it become smaller

Open wcyjerry opened this issue 3 years ago • 1 comments

Hey, I met this problem I dont know whether it is common or it is a bug. eg. When I monitoring the gpu-usage, in the first place it will up to 11Gb, and after a small time, it drops to 6G and become constant while the whole training process. But I havnt met this problem in other frameworks or my own project before.

Jul 25 '22 07:07 wcyjerry

I am not sure what happens behind it.

Will it cause an out-of-memory issue? or will it stop the training?

Jul 26 '22 11:07 xinntao