HPSG-Neural-Parser icon indicating copy to clipboard operation
HPSG-Neural-Parser copied to clipboard

Punctuation must be separated from words?

Open nangeblog opened this issue 6 years ago • 1 comments

Punctuation must be separated from words?This is inconvenient

sentences: 1 no , it was n't black monday . 2 no , it was n't black monday. 3 no, it was n't black monday . 4 no, it was n't black monday. 5 some " circuit breakers " installed after the october 1987 crash failed their first test , traders say , unable to cool the selling panic in both stocks and futures . 6 some "circuit breakers" installed after the october 1987 crash failed their first test, traders say, unable to cool the selling panic in both stocks and futures.

the head: 1 [7, 7, 7, 7, 7, 7, 0, 7] 2 [6, 6, 6, 6, 6, 0, 6] 3 [6, 6, 6, 6, 6, 0, 6] 4 [5, 5, 5, 5, 0, 5] 5 [4, 4, 4, 12, 4, 4, 6, 11, 11, 11, 7, 0, 15, 15, 12, 18, 18, 12, 18, 12, 22, 20, 25, 25, 22, 25, 28, 26, 28, 28, 12] 6 [2, 10, 2, 2, 4, 9, 9, 9, 5, 0, 12, 10, 16, 16, 16, 10, 18, 16, 21, 21, 18, 21, 24, 22, 24, 10]

nangeblog avatar Sep 03 '19 02:09 nangeblog

it is a great way by adding a function or class like BertTokenizer

nangeblog avatar Sep 03 '19 03:09 nangeblog