perplexity.compute - Add args to pass model and tokenizer objects instead of `model_id`

[albernar]

Hello, I am new to the project, so I hope my request is a sensible one.

It looks like when we use the compute method for Perplexity, we can only specify a model through its model_id, which is used also to load the tokenizer.

Is there any particular reason why it is not possible to pass as arguments to the method the model object and the tokenizer object themselves instead of the model_id? I think that could help, for example when the tokenizer has some user-added tokens or when the model to be used is fine-tuned (e.g., currently, a PeftModel could not be used, but passing the model object itself that should not be a limitation anymore).

Thanks for your answer and for your amazing work.

[KangkangStu]

Hello, I am new to the project, so I hope my request is a sensible one.

hello，I am also a new to the project.I'd like to ask if you've implemented an assessment of the perplexity of the local model