microsoft
Enhancing quality - Recovery settings
As mentioned in the paper, key concepts might get omitted either corrupted by the compression, in a way that the GPT can't process the compressed prompt.
You mention also there is an approach to optimize around this issue; could you share details on the corresponding configuration options in the Python implementation?
In the attached image, I've tested the GPT confidence degradation according to compression effects on the qasper_e subset of the LongBench benchmark.
Wrong answers/no answer possible:
- Regular GPT-4: %45.36 e.g. without prompt compression (GPT-4 seems to "give up" frequently on longer queries)
- Compressed prompt by LLM Lingua, target_token=200: 63.93%
- Compressed prompt by LLM Lingua, target_token=400: 60.66%
Hi @synergiator, thank you very much for your interest in LLMLingua and for sharing the detailed experimental results. They are very helpful to us.
You can find the recovery function at https://github.com/microsoft/LLMLingua/blob/main/llmlingua/prompt_compressor.py#L922. However, I suspect the increase in the no answer ratio in these cases is due to the loss of necessary information. I'm curious whether you used LongLLMLingua or LLMLingua; if it was LLMLingua, the loss of valuable information might be more significant, especially with a high compression ratio of about 10x-20x.
Nevertheless, we greatly appreciate your experiments and conclusions.
