saber
saber copied to clipboard
Switch token alignment to SpaCy
Currently, to align BERT tokens to original tokens (before BERT tokenization) we use some code I grabbed from the official BERT repo.
SpaCy has introduced functions specifically for aligning two tests tokenized with different tokenizers. Switch to this!