Why get_special_tokens_mask appending a [1] at the end while build_inputs_with_special_tokens does not append a [SEP] at the end for a single input sequence ?

[zhihuacc]

Hi, I found in this line[build_inputs_with_special_tokens](url) the returned list is appended a [1] at the end for a single input sequence, while the returned list [here](url) is NOT appended a [SEP] for the same case. Why is that ?

[LiJunnan1992]

We remove [SEP] for a single sentence input because it has negligible effect on pre-training.

[zhihuacc]

But why get_special_tokens_mask still appends a [1]. I thought this [1] is for [SEP], right ?

[LiJunnan1992]

Yes you are right, I have modified the code so that the [1] is not appended. Thank you!