LLMLingua icon indicating copy to clipboard operation
LLMLingua copied to clipboard
verified publisher icon microsoft

LLMLingua doesn't work on CPU as device_map

Open MrTBH opened this issue 1 year ago • 2 comments

Hi. I've been trying to use LLM Lingua on CPU Linux Machine. When I start by python file. it just stops like the attached screenshot Screenshot_1

The code I'm using is as simple as following:

PromptCompressor(device_map="cpu")

MrTBH avatar Feb 03 '24 05:02 MrTBH

Hi @MrTBH, it looks like out of memory. You might want to try using a smaller model like lgaalves/gpt2-dolly or a quantized version of the model.

iofu728 avatar Feb 05 '24 12:02 iofu728