doublearray icon indicating copy to clipboard operation
doublearray copied to clipboard

stringToUtf8Bytes may return null

Open alpaca-tc opened this issue 3 years ago • 1 comments

Fix TypeError: Cannot read properties of null (reading 'length'). I have not been able to reproduce the bug but it seems that stringToUtf8Bytes may return null.

stringToUtf8Bytes returns null ↓ https://github.com/takuyaa/doublearray//blob/461f142db0e9111672e5528770546052a7a1ec30/doublearray.js#L685-L685


By the way, the error occurred while processing a string containing emoji like '😂' with kuromoji.

TypeError: Cannot read properties of null (reading 'length')
    at DoubleArray.commonPrefixSearch ((...)/doublearray/doublearray.js:559:36)
    at ViterbiBuilder.build ((...)/node_modules/kuromoji/src/viterbi/ViterbiBuilder.js:47:36)
    at Tokenizer.getLattice ((...)/node_modules/kuromoji/src/Tokenizer.js:126:33)
    at Tokenizer.tokenizeForSentence ((...)/node_modules/kuromoji/src/Tokenizer.js:81:24)
    ...

alpaca-tc avatar Dec 01 '21 13:12 alpaca-tc

@takuyaa Today, I confirmed it happens when using 🍪.

alpaca-tc avatar Mar 11 '22 08:03 alpaca-tc