Tutorial 6: byte pair encoding

[brand17]

As far as I know - byte pair encoding is used in the original transformer model. Why don't you use it ?

[bentrevett]

It's true, BPE is used in the Transformer and is now quite common in NLP. I didn't use it as I mainly wanted to focus the tutorials on the models themselves and not the pre-processing.

I believe Torchtext has an interface to Sentencepiece which should allow for BPE to be used easily. I'll look into adding it to the tutorials.