LLMLingua icon indicating copy to clipboard operation
LLMLingua copied to clipboard
verified publisher icon microsoft

use other quant formats

Open zba opened this issue 2 years ago • 1 comments

Is it would be hard to use exl2 for same purpose ? Or openai compatible api ?

zba avatar Dec 30 '23 20:12 zba

Hi @zba,

Thank you for your interest and support in LLMLingua. I believe there are no block issues with using the exl2 format. You can try replacing the code at LLMLingua Prompt Compressor with ExLlamaV2.

For the OpenAI format, you can use the latest API to obtain log probabilities and set max_tokens to 0. This will help you get the log probabilities for the prompt portion.

iofu728 avatar Jan 03 '24 11:01 iofu728