Drop prompt from output is determined in an unclear manner in HFModel

[bendavidsteel]

Dropping the prompt from the model output is necessary to be able to correctly retrieve an output from the Prediction object, but this is only done in HFModel when a ValueException is thrown upon model creation for unclear reasons. It would be useful to either have this property setable by the class user, or have the reason for it's current set up be explained?

Line 79 onwards in hf.py for reference.

[okhat]

Just to clarify, are you saying there's an issue in HFModel? Or HFModel is correct and the rest aren't?

(In general, HFModel is very clunky and needs work. For local models, we usually recommend TGI clients for now.)

[bendavidsteel]

That there's an issue in HFModel. If that's the case I can accept that! Will maybe make a PR with my local changes to get it usable.

[okhat]

That would be really useful! We'd love to merge your fixes.

That said, I recommend looking into using TGI. It's a lot faster, more customizable, and much more stable than HFModel.