DialoFlow icon indicating copy to clipboard operation
DialoFlow copied to clipboard

about data tokenizer

Open JYlsc opened this issue 3 years ago • 0 comments

hello, i see tokenizer seq in paper is : [u1] [C] [u2] [C] [res] [C]

but tokenizer in code dataset is : [speaker1] [u1] [eos] [speaker2] [u2] [eos] [bos] [res] [eos]

Is there any difference between the two? which works best

JYlsc avatar Mar 03 '22 08:03 JYlsc