Angainor Development
Angainor Development
No idea about chinese myself, but maybe you could check and give guanaco a try https://huggingface.co/KBlueLeaf/guanaco-7B-leh https://github.com/KohakuBlueleaf/guanaco-lora This is a fork of this repo addressing chinese and more.
Please provide more context. finetune or inference? precise command launched? gpus type and count? clean py env or some other one from alpaca serve? Also see the debug hint the...
@ElleLeonne Did you try with set inference_mode=False in the PeftConfig ? line 25 of your minimal code, try adding add peft_config.inference_mode=False
Make sure you use the latest code. This could be because you have an older code, not compatible with current peft.
@carmocca Is this of interest? I would have some related suggests and PR, but I prefered to keep this one as small as possible for a start.
@carmocca Sure, just done!
Sorry! I'll process them as well asap.
Done for all finetune/ scripts. Both scripts in pretrain/ use a different loop and rely on iters, with no explicit steps.
> What do you think of this implementation? Thanks for this! I was trying to get batch inference working myself, hoping for a lower inference time. I used your code,...
Thanks, Yes I did the real token counts. token/s is coherent with generation time. A single generation in my case is 5 sec, while a batch of 4 (even 4...