Initial commit of reranking with cross-encoder cookbook

[colin-openai]

trafficstars

[juntang-zhuang]

Hi, it looks good to me in general. One corner case I'm thinking is that the model might output the next most probable token not in ['yes','no'] (I expect this to happen at an extremely low frequency). In this case, we might need to take out the logits for ['yes','no'] then perform a softmax as post-processing.

[ted-at-openai]

Hi, it looks good to me in general. One corner case I'm thinking is that the model might output the next most probable token not in ['yes','no'] (I expect this to happen at an extremely low frequency). In this case, we might need to take out the logits for ['yes','no'] then perform a softmax as post-processing.

Two options here, since you cannot grab the full distribution:

• Use logit_bias to massively upweight the ' yes' and ' no' tokens (better if you need both tokens)
• Use echo=True to always get the logprob for ' yes' (better if you just need ' yes')

[colin-openai]

@ted-at-openai @juntang-zhuang updated with logit bias for Yes and No, seems to work well. Let me know if there is any other feedback, otherwise ready to merge.

[colin-openai]

@ted-at-openai thanks for reviewing, I've left open the ones that need an extra check on how I've responded - once you're happy I'll merge.

[ted-at-openai]

Looks great. Only two final recommendations:

• Use a big logit_bias (e.g., 100 instead of 1; 1 is small enough that a lot of other tokens will still come in, which explains why you saw 'No' outranking ' Yes' sometimes)
• Rerun all the cells at the end so they're in numerical order (without any distracting jumps from 5 to 30 where someone wonders what the author was doing for those 25 steps in between :) )