合并为hf版本，能否直接作为alpaca_lora的base_models

[quantumsquirrel]

想了一个使用GPU的思路，合并模型权重的时候选择合并为hf版本， 直接作为原生alpaca_lora的base_models， 配合hf上面的lora_weights 调用原生的Alpaca-LoRA进行推理。 个人测试了一下，载入模型是成功的，但是载入以后就报错了，有没有相同思路的朋友尝试过，不知道是不是转模型的时候转错了。

I came up with an idea to use GPU, merging the model weights by selecting the HF script during the merge, and directly using them as the base_models of the native Alpaca-LoRA, along with the lora_weights on HF, to perform inference. I tested it myself and was able to load the model successfully, but encountered an error after loading. I’m wondering if anyone else has tried this approach, and if it’s possible that the error was caused by an incorrect model conversion.

[airaria]

合并了之后就不再需要lora权重了，不用再提供lora_weights了吧？

[ykk648]

大模型想着用CPU推理简直就是在倒退，cver昨天入门nlp，今天也在尝试gpu推理

[airaria]

想了一个使用GPU的思路，合并模型权重的时候选择合并为hf版本， 直接作为原生alpaca_lora的base_models， 配合hf上面的lora_weights 调用原生的Alpaca-LoRA进行推理。 个人测试了一下，载入模型是成功的，但是载入以后就报错了，有没有相同思路的朋友尝试过，不知道是不是转模型的时候转错了。

I came up with an idea to use GPU, merging the model weights by selecting the HF script during the merge, and directly using them as the base_models of the native Alpaca-LoRA, along with the lora_weights on HF, to perform inference. I tested it myself and was able to load the model successfully, but encountered an error after loading. I’m wondering if anyone else has tried this approach, and if it’s possible that the error was caused by an incorrect model conversion.

更新了推理脚本 scripts/inference_hf.py，可参考。