Fast-BEV icon indicating copy to clipboard operation
Fast-BEV copied to clipboard
verified publisher icon Sense-GVT

Huge memory consumption during training

Open LaCandela opened this issue 1 year ago • 1 comments

I noticed that during training the process consumes lots of CPU memory. For example, if I use 8 workers with batch size of 45 the memory consumption is more than 50GB and it is slowly increasing up to 60GB when my system shuts down. Has anyone noticed this before? What is the memory intensive part of the training pipeline? Is it related to some inefficient mmcv operation?

I start the training with this command: CUDA_VISIBLE_DEVICES=0 python tools/train.py --config configs/fastbev/exp/paper/fastbev_m0_r18_s256x704_v200x200x4_c192_d2_f4.py --work-dir /Fast-BEV/temp

Could it be a problem that I don't use slurm?

LaCandela avatar Apr 23 '24 13:04 LaCandela

Have you up the project ? can you provide me steps to build and run properly ?

venkataramanas1 avatar May 13 '25 12:05 venkataramanas1