南栖
南栖
> @NilanEkanayake So there are some pre converted Qwen models on HuggingFace if you search for "qwen llama". > > In terms of Qwen 1.5 / 2 - if it's...
> @songkq Oh it should be supported if you use Llama-Factory's llamafy script. Ie maybe try https://huggingface.co/models?search=qwen%20llama. On the other hand, if some don't exist, you can try out Llama-Factory's...
> > @songkq Oh it should be supported if you use Llama-Factory's llamafy script. Ie maybe try https://huggingface.co/models?search=qwen%20llama. On the other hand, if some don't exist, you can try out...
我觉得你可以冻结预训练模型的层，可以保留最后一层用来下游微调训练，这样不会改变模型整体的学习内容
转成huggingface后效果咋样，会有损失吗？
我的建议是lora微调代码用[tloen](https://github.com/tloen) / [alpaca-lora](https://github.com/tloen/alpaca-lora)斯坦福官方的lora微调代码，这个代码 我也用了，微调没法用int8微调，tesla A40 微调lora爆显存，但是斯坦福alpaca可以int8 lora微调 13G显存就能微调70B
是的,我也发现这个问题了,所以后来换了斯坦福的微调代码,单块tesla a100 可以微调llama 650亿 [tloen](https://github.com/tloen) / [alpaca-lora](https://github.com/tloen/alpaca-lora)
嗯，明白了，期待13b的中文模型。
Same problem.
> I install auto_gptq 0.7.0 Change auto_gptq version