spacyface icon indicating copy to clipboard operation
spacyface copied to clipboard

Pass additional arguments to transformers tokenizer

Open happen2me opened this issue 2 years ago • 0 comments

There are cases when the default options of transformers tokenizer don't meet our demands, but the tokenizer is wrapped inside Aligner, i.e. we wish the sentence_to_input function to also return special tokens mask. This modification passes additional tokenization parameters to transformers prepare_for_model().

happen2me avatar May 09 '22 22:05 happen2me