UNICODE fixes (clipboard, window title, file dialogs, paths, font glyphs, ...)

[nixbody]

Adds UNICODE support for CFFI related things on HXCPP and HashLink targets.

[player-03]

Hey @restorer, can you take a look at this one and tell us how it compares to #1391?

[player-03]

@joshtynjala Can you resolve the conflicts in ExternalInterface.cpp? It looks like you were both trying to solve the same problems but solved them in slightly different ways.

[nixbody]

I'd like to discourage from using std::wcstombs even on Windows because the encoding of the resulting string is locale-dependent and while some more recent versions of Windows 10 (and newer) allow to set locale to UTF-8, it's probably never the default system locale. Windows API uses UTF-16 (UCS-2 before Windows 2000) for file names and it's quite possible to have files with names that cannot be translated to a narrow string using the default system locale.

[joshtynjala]

. @joshtynjala Can you resolve the conflicts in ExternalInterface.cpp? It looks like you were both trying to solve the same problems but solved them in slightly different ways.

[nixbody]

In order to resolve the conflicts some maintainer needs to decide which one of those two similar functions (hl_wstring_to_utf8_bytes() vs wstring_to_vbytes()) they wish to keep. Mine is locale-independent so I think it's better but I really wish for this PR to get merged so I go either way.

[tobil4sk]

which one of those two similar functions (hl_wstring_to_utf8_bytes() vs wstring_to_vbytes()) they wish to keep

I've opened up a another PR #1665 to clean things up so hopefully we can avoid these conversions altogether.

[tobil4sk]

Actually, it looks like there are a few other places where these conversions are still required, so might need to keep one of these functions around anyway, unless those wstrings can be avoided as well.

[tobil4sk]

In order to resolve the conflicts some maintainer needs to decide which one of those two similar functions (hl_wstring_to_utf8_bytes() vs wstring_to_vbytes()) they wish to keep.

Would it be possible to decide on one of these so that these unicode issues can be fixed? Maybe @nixbody's one because it is locale independent?

(For reference, the commit that introduced the conflicts was 95411acb8c6edcecbd80e756b692dff0f55c4571)

[tobil4sk]

I've forked and resolved the merge conflicts in a separate branch, preferring hl_wstring_to_utf8_bytes over wstring_to_vbytes: https://github.com/tobil4sk/lime/tree/cffi-unicode-fixes

This is quite an important fix. If there are no objections, these changes can be pushed here and this could be merged? Then, once things are working finally, in future we can look at cleaning up the conversions in #1665.

[player-03]

I didn't have permission to push your branch from my machine, so I redid the changes in the web editor. Let me know if I got anything wrong.

[tobil4sk]

I added a commit here to put it in sync with my branch.

[tobil4sk]

There is a conflict now in src/lime/text/Font.hx because of #1742. I think it might be fine to keep that file as it currently is in develop, it seems to me that both changes deal with the same issue? @nixbody

[player-03]

I think I'd prefer the version here, without the unsafe cast.

[tobil4sk]

I think I'd prefer the version here, without the unsafe cast.

You're right, looks like this is the preferred way to handle hl.NativeArray -> Array conversions with hashlink, e.g. in hlsdl:

https://github.com/HaxeFoundation/hashlink/blob/783cd2a785b2e6f38e6b01e1b3bf5bdc19fa1173/libs/sdl/sdl/Sdl.hx#L141-L144