byaldi Model is not being offloaded from VRAM

I am trying to run the model in Jupyter notebook.

In the above iteration I haven't initialized the model.

2. Now I run the cell the model is loaded and it is showing 6GB of vram occupied right.

3. Now when I run the cell again the vram usage is doubled. 4. In the consequent runs the model is not occupying more than 12GB but what's interesting thing I have observed is when I am running that inside a loop for suppose I want to create an Index for each file I have, I don't have any other option than do this but this is causing the model to give me vram issues. How do I remove them from vram, I tried torch cuda cache free, tried to delete the variable none isn't working for me. Can you please help or is there something I am doing wrongly ?

Oct 08 '24 12:10 nishithshowri006

Could you provide your notebook as a Colab notebook so I can more easily reproduce the exact issue? Thank you!

Nov 11 '24 07:11 bclavie

Hey colab notebook this is just a basic observation I had. You might have more understanding than me, I added comments in the notebook on what I observed.

Nov 11 '24 08:11 nishithshowri006

I believe, I can help with this issue. Could you assign the issue to me?

Nov 14 '24 05:11 DebopamParam