semantra icon indicating copy to clipboard operation
semantra copied to clipboard

Published 20 hours ago • verified publisher icon freedmand

Garbled characters when using OpenAI models to embed Chinese

Open ted-b opened this issue 1 year ago • 3 comments

Garbled characters when using OpenAI models to embed Chinese. But using --transformer-model intfloat/multilingual-e5-base it's normal. endoing utf-8 What's the promble? 1685634411892

ted-b avatar Jun 01 '23 15:06 ted-b

Hmm weird that the title is showing up just right. Might be a problem with indexing? But not sure why E5 works.

如果你可以分享一下有问题的文件的话，我可以尝试一下复现 (I can try to replicate the problem if you can share the file)

yych42 avatar Jun 01 '23 15:06 yych42

I managed to confirm the issue. The problem seems to be in xxx.tokens.json, where "\ufffd" pops up a lot when using openai.

Imo openai is not that much superior than e5 when it comes to search in Chinese, so not sure if this should be fixed now or after there's a plan for v0.2---this will be up to @freedmand

yych42 avatar Jun 01 '23 22:06 yych42

Thanks for your reply. Looking forward to V0.2 improving this issue.

ted-b avatar Jun 02 '23 13:06 ted-b