growing gpu usage during training

[jeffhwang02]

@lkeab I am try to train the teatime, but it shows continuous growing usage of gpu memory, it will stop at the time of full gpu memory.

How to avoid this issue?

[txlu4]

even though the command lines seems don't move anymore but actually it is still working, it just needs a very long time to train