llm-foundry
llm-foundry copied to clipboard
mosaicml
Generated sample equals to input samples
I have trained 125m MPT on some small dataset, my generated inputs via inference/hf_generate.py (before this converted from composer to HF, and it's gives me some value from eval/eval.py with winograd) are same with input samples, do u know what kind of problem it could be?
Hi @germanjke , could you clarify what data you have been finetuning on? Are you running the Quickstart instructions? If so, the outputs are not meant to be good, it's just meant to be a tour of the different scripts available in the LLM Foundry and take you from data -> model -> generations.
It would also help greatly if you could attach the commands and outputs you are seeing.
Hi, I am using the 'inference/hf_generate.py' with 'mpt-7b-instruct' (inference only) but also getting generated output which is the same as input for certain prompts. Also the answers seem to be more of a text completion, and are not similar to the instruct demo responses. Could you clarify why this is happening, and what are the recommended changes for using this model to be able to be similar to the demo? I have set the temperature, top_p and top_k as in demo already.
Hi @kochkinaelena , could you provide a bit more info such as the prompts that you are using, or ideally the exact command python hf_generate.py ... you are running?
For the mosaicml/mpt-7b-instruct model specifically, I believe it may be best to follow the instruction format that it was finetuned with, you can see an example here of how we prompt the model in our HF space: https://huggingface.co/spaces/mosaicml/mpt-7b-instruct/blob/main/quick_pipeline.py#L8-L23. This might lead to better generations!
