Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface)

[HinTak]

There is no need to restrict to ascii file names?

Cannot marshal: Encountered unmappable character.   at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface)

I see it is set in the DllImport marshalling to

CharSet = CharSet.Ansi, BestFitMapping = false, ThrowOnUnmappableChar = true

Here is the list of routines with the above set:

        internal static extern Error FT_New_Face(IntPtr library, string filepathname, int face_index, out IntPtr aface);
        internal static extern Error FT_Attach_File(IntPtr face, string filepathname);
        internal static extern Error FT_GetFile_From_Mac_Name(string fontName, out IntPtr pathSpec, out int face_index);
        internal static extern Error FT_GetFile_From_Mac_ATS_Name(string fontName, out IntPtr pathSpec, out int face_index);
        internal static extern Error FT_GetFilePath_From_Mac_ATS_Name(string fontName, IntPtr path, int maxPathSize, out int face_index);
        internal static extern Error FT_Get_BDF_Charset_ID(IntPtr face, out string acharset_encoding, out string acharset_registry);
        internal static extern Error FT_Get_BDF_Property(IntPtr face, string prop_name, out IntPtr aproperty);
        internal static extern Error FT_Get_CID_Registry_Ordering_Supplement(IntPtr face, out string registry, out string ordering, out int aproperty);
        internal static extern IntPtr FT_Get_Module(IntPtr library, string module_name);
        internal static extern Error FT_Property_Set(IntPtr library, string module_name, string property_name, IntPtr value);
        internal static extern Error FT_Property_Get(IntPtr library, string module_name, string property_name, out IntPtr value);
        internal static extern Error FT_Property_Get(IntPtr library, string module_name, string property_name, ref int value);

PropertyGet/Set never have non-ascii arguments, but file names and font names do!

[HinTak]

Argh, this is a more difficult issue than I thought. Mono apparently treats CharSet.Ansi as utf8, which is why I never saw it until now.

I am seeing it because I am running SharpFont under wine with the genuine MS dotnet runtime at the moment (it is just a special thing that I need to do now, instead of using native mono). The behavior with CharSet.Ansi is as documented but undesirable - freetype itself on windows can open font files with non-ascii file name or font names, just treat them as null terminated c strings. Using CharSet.Ansi is uncessarily restrictive, but CharSet.Unicode probably won't work on windows, since freetype does not really expect arguments to come in as UTF16LE's (I assume that's what is meant by marshalling as "CharSet.Unicode").

[Robmaister]

Yikes, this looks like a pretty complex one. I suppose we could try to manually encode UTF-8 and have the extern functions take IntPtrs instead. Does FreeType expect UTF-8 for those APIs on all platforms?

[HinTak]

Freetype treats the input as raw bytes, does not really apply any encoding to it - as long as it is null-terminated c byte arrays. I can't see a quick way fixing this. One possibility is you open the file in c# and pass the file content as a memory block.

[Robmaister]

There are already Face constructors that do this for FT_New_Memory_Face - right now two constructors:

• public Face(Library library, byte[] file, int faceIndex)
• public Face(Library library, IntPtr bufferPtr, int length, int faceIndex)

I can also add an extra overload that takes some type of stream (FileStream? MemoryStream?) and reads all bytes, chaining to the byte[] constructor.

Another thing we can do is adapt a C# stream to FTStream so that it's easy to call Library.OpenFace.

[HinTak]

I guess what I like, is for public Face(Library library, string path) and public Face(Library library, string path, int faceIndex) to open the file name on the C# side and use FT_New_Memory_Face instead of calling FT.FT_New_Face with a non-ascii argument (and fail), and/or optionally doing the former if the filename is non-ascii.

[HinTak]

Back-ported streaming support https://github.com/HinTak/SharpFont/commit/ff46f019f89c24dfa196b0c7e283371c26acdc33 and https://github.com/HinTak/SharpFont/commit/c574260a183d657bca59c1072dc3c141190993e6 and modified my caller to new Face(library, stream, faceindex, false).

Running on English name was successful, I have not tried non-ascii name yet.

[HinTak]

I'll be testing the backport soon, so assuming that's successful, this can be closed.

[HinTak]

The streaming support seems to lose the file handle once in a while - like one test out of 50... needs better looking at.