pycantonese
pycantonese copied to clipboard
Segmenter removes space of English words in code-mixed sentence
Describe the bug Segmenter removes space of English words in code-mixed sentence, for example this sentence:
這是Career Centre
To reproduce Here is the code:
import pycantonese
from pycantonese.word_segmentation import Segmenter
segmenter = Segmenter()
pyseg = pycantonese.segment("這是Career Centre", cls=segmenter)
for word in pyseg:
print(word)
The output is:
這是
CareerCentre
Expected behavior The expected output is:
這是
Career Centre
or
這是
Career
Centre
System (please complete the following information):
- Operating System: macOS Sonoma 14.0 (23A344)
- PyCantonese version:
3.4.0
After a dig in the old issues, I thought this issue was fixed in https://github.com/jacksonllee/pycantonese/issues/32#issuecomment-1268983221, but it isn't.
主要係因為呢個 https://github.com/jacksonllee/pycantonese/pull/35 未解決所以一直都未發佈更新。
I guess I have to wait then.
You can replace the space with some uncommon punctuations, such as "▁". And then skip it.
https://github.com/pengzhendong/g2p-mix/blob/dd19bee513cc13230c41ef66e479de695afa0e2c/g2p_mix/g2p_mix.py#L43