Handle (remove) zero-width-space

Open jaggzh opened this issue 1 year ago • 1 comments

We currently end up converting zero-width-space (zws) U+200B / \xe2\x8\x8b to an invalid sequence (FD BF BF BD A3 AC). We (chatgpt and I) added code to just remove these, (in the same place that we swap the non-breaking space with space).

Dec 25 '24 00:12 jaggzh

Hi, sorry for the delay. I'm not sure I understand the patch, you are saying there is a bug in the handling of zws somewhere else, so to work around it you get rid of all zws early on? Do you have a link or some other reproducer for the issue?

May 11 '25 22:05 eafer