GPU Memory-Usage goes higher during Trainning.

[yuanryann]

Hi Dr.Zhou, Firstly, Thank you very much for your excellent work!!!. I run trainning on my Nvidia-A4000 GPU，GPU Memory-Usage goes higher every epoch. Some hparams are changed: the train_batch_size and the val_batch_size set as 64. parallel :True,num_workers:6,pin_memory: false Epoch 0, the GPU Memory-Usage takes up 11000Mib, but Epoch 30, it takes up 15577Mib. Could Anyone help me deal with this issue.

the hparams.yaml shown as below: historical_steps: 20 future_steps: 30 num_modes: 6 rotate: true node_dim: 2 edge_dim: 2 embed_dim: 64 num_heads: 8 dropout: 0.1 num_temporal_layers: 4 num_global_layers: 3 local_radius: 50 parallel: true lr: 0.0005 weight_decay: 0.0001 T_max: 64 root: /home/com0179/AI/Prediction/HiVT/datasets train_batch_size: 64 val_batch_size: 64 shuffle: true num_workers: 6 pin_memory: false persistent_workers: true gpus: 1 max_epochs: 64 monitor: val_minFDE save_top_k: 5