Sovit Ranjan Rath

Ok. Can you let me know the GPU model and the memory usage during the training loop?

Oh. I too do all experiments on an RTX GPU. But I have never faced this. May take some time to figure this out if I am unable to reproduce...

Ok. That's interesting. Because while creating the codebase, I took care of checking all places where memory overflow may happen and I never faced the issues. Anyways, thanks for reporting...

I tested it again and did not face any issues. The GPU memory usage remained exactly the same during the validation loop and did not increase at all.

Hi. At the moment, during the training/validation it cannot be changed as it used pycocotools and it is quite difficult to do so.

Hello @yochju I am not sure how it will behave with 10 GPUs. I have only tested it with 2 GPUs, but I guess you can safely run training on...

@yochju Actually, by mistake I gave `--nproc_per_node=2` for 4 GPUS, it should have been 2. It's interesting that it is working with `--nproc_per_node=2`. I will try to check this on...

Trying to fix this.

I tried a few experiments with 2 GPUs first just to ensure that there are no issues. However, it will be near this weekend that I will try to run...

Hello. Can you please provide the command that you are using?