Include Prompts in Evaluation?

[haikangdeng]

Hi, I'm curious about the intuition to include or not include prompts during evaluation. In your toxicity evaluation, only the generated sequence is evaluated. However, in your sentiment evaluation, prompts are prepended to the generated sequences, and the entire sequence is evaluated. Would it be better to keep the settings consistent in these experiments? Which do you recommend, keep or remove prompts? Thanks

Toxicity

https://github.com/alisawuffles/DExperts/blob/30e2af536445a43977037f927039517175f58e8a/scripts/run_toxicity_experiment.py#L171-L173

Sentiment

https://github.com/alisawuffles/DExperts/blob/30e2af536445a43977037f927039517175f58e8a/scripts/run_sentiment_experiment.py#L165-L171