GPU memory requirement

[Raspberry-beans]

Thanks for the great work!

I will be replicating your approach to fine tune SAM on custom few shot medical images. The problem is I have only 8GB GPU memory available from my university.

Would it be possible to replicate your approach having only 8GB memory space. If not, is there any way I can do fine tuning requiring less than 8GB memory e.g using batch_size of 1, reducing size of prompt layer or using SAM Vit-B image encoder etc

Your suggestions will be appreciated.

Regards, Muhammad

[Qsingle]

Thank you. You can try the ViT-B as the backbone and set the batch size to 1. If the memory is not enough. You can also try to downsample the input and resize the position embedding. - You can follow our anohter repository.

[Raspberry-beans]

Thanks a lot for your response. I am also thinking to first fine tune mask-decoder only (keeping image and prompt encoder frozen) for few custom images. Would you think this would be possible in my case as SAM mentioned that their mask decoder is lightweight?

Have a nice dary!

[Qsingle]

Yeah, the decoder is lightweight.