slow inference with peft model

[wwh0411]

I use the code from fingpt benckmark to evaluate my peft model on datasets：fpb/fiqa/tfns respectively. However the inference speed is very slow, and increasing batch size doesn't help with this issue. It seems that when i increase the batch size, the inference speed per iteration increases consequently. Could u please help with this issue?

[BruceYanghy]

Which GPU device are you using?

[wwh0411]

I‘m testing the model on 2 RTX 3090s

[Weiyao-Li]

Could you try google colab with A100 or TPU. Did you make any progress or still having any issue now?