Memory Overflow Issue

[plpalmer]

Hello,

i have noticed that used system memory increased over the course of the training. The training of the model than crashes when the system memory is completely filled up. It usually takes multiple Epochs. Have you noticed this issue before? Do you have an explanation, or some idea what could cause this issue and what i should look further into? I use the model with my own data, but have also noticed this issue when used with your provided Waymo data.

Thanks for your help in advance