Is is possible to run the pre-trained model with rtx 2060 6g?

[dream-desktop]

My computer has the rtx 2060 6G, the results say my gpu is lack, is there any way to run the phoenix model. thank you.

[zhjohnchan]

Hi @dream-desktop,

Thanks for your attention. Currently, 6G is not sufficient for running Phoenix. We will work on the Quantization version.

Or you can try Phoenix using cpu:

python -m llmzoo.deploy.cli --model-name FreedomIntelligence/phoenix-inst-chat-7b --device cpu

which requires ~30GB of CPU memory.

Best, Zhihong

[wabyking]

Please check if this helps: https://gist.github.com/rain-1/8cc12b4b334052a21af8029aa9c4fafc @dream-desktop

[wabyking]

It is for LLaMA based models