spacyface
spacyface copied to clipboard
Pass additional arguments to transformers tokenizer
There are cases when the default options of transformers tokenizer don't meet our demands, but the tokenizer is wrapped inside Aligner, i.e. we wish the sentence_to_input
function to also return special tokens mask. This modification passes additional tokenization parameters to transformers prepare_for_model()
.