Diff breaks unicode characters for emojis

[orromis]

I'm working message log in our app. We want to show diff of changes any user made on text posts. Those posts may include emoji characters but diff_match_patch replaces those characters with � character (but only if they changed in the text).

The behaviour can be reproduced here: https://neil.fraser.name/software/diff_match_patch/demos/diff.html

Paste 😉and 😀into textareas and compute the diff.

Why is this happening?

[mcataford]

Looks like the diffing doesn't consider whether a character is non-ASCII and breaks unicode emojis in pieces if they are different but occupy the same space. That would result in some unknown character � after all is said and done. Assuming it's conversion related.

Looking into it out of curiosity.

[josephrocca]

Looks like @yetanotherape has solved it for their PHP fork:

• https://github.com/yetanotherape/diff-match-patch/issues/9
• https://github.com/yetanotherape/diff-match-patch/commit/3e7b0241a06b20ad348c1d35f77204d02ec346bc

The two current attempts at solving it in this repo have both had complications:

• https://github.com/google/diff-match-patch/pull/13
• https://github.com/google/diff-match-patch/pull/69

[ndvbd]

Any updates here?

[josephrocca]

@ndvbd I ended up "solving" it by just escaping all the special unicode stuff with text = encodeURI(text) before saving the text (and using decodeURI(...) to undo it, of course). Bit of a hack but it works for my use case.

[dmsnell]

Check out #80, as referenced above @ndvbd's comment. It should handle all the surrogate pairs properly.