How to run in linux machine (CPU without GPU)

Open gayuoptisol opened this issue 1 year ago • 2 comments

Is there any method to run LLMLingua in linux cpu machine. I am trying to load this using: from llmlingua import PromptCompressor llm_lingua = PromptCompressor(device_map="mps")

but it taking so much amount to load.

Jan 24 '24 08:01 gayuoptisol

Hi @gayuoptisol,

You can set the device_map to “cpu” as follow,

llm_lingua = PromptCompressor(device_map="cpu")

Jan 24 '24 09:01 iofu728

hi :-) is there a rule of thumb - how much GPU memory is required depending upon model/processed content, and how does this translate to required number of CPUs and their memory?

Feb 19 '24 12:02 synergiator