Running out of memory (2070, 8 GB)

[kallewoof]

Updating the README to indicate the amount of VRAM necessary would be very helpful as it's not clear right now if I'm way below required amount or just snuggling it.

[limapedro]

it seems that it needs 12 GB, I have 6 GB, i'm looking to run on CPU as of it now.

[C0untFloyd]

The latest commits helped a lot with memory management. I made my own fork (with Gradio GUI) and develop/run this on a GTX 2060 Super with only 8 Gb. Perhaps when switching to use small models it might even run with 6 Gb.

[limapedro]

can you point me a link to the smaller models?

[C0untFloyd]

They are downloaded & used when setting the Environment-Variable SUNO_USE_SMALL_MODELS or just use my fork where it is a cmdline-argument 😉

[gkucsko]

Closing for now as this should be possible using environment variables and preload_models.

[kallewoof]

@gkucsko Ideally, this would serve as a nudge to provide RAM details in the README. Not going to argue though.

[gkucsko]

thanks yea we are due for some documentation updates very soon