Special Characters in default fonts are missing

[sirtobi79]

I did some tests on localization with the default font. The German characters "ß,ä,ü,ö" are missing.

Console output:

3 warnings:

Font is missing glyph for character ß (Unicode number 223) Font is missing glyph for character ä (Unicode number 228) No further warnings about missing glyphs will be reported for this font (to avoid slowing down the application by flooding the log with warnings)

Scene URL: "file:///C:/NetBeans/mv3204x3dforwebauthors/assignments/DirectedStudy/Tobias/Scenes/HelloGermany.x3d". snip

[michaliskambi]

Indeed, that is a known issue.

Right now in the underlying Castle Game Engine we don't have a way to load additional font glyphs "on demand". We could just extend the current font with more and more characters (see https://castle-engine.io/manual_text.php ), but as view3dscene is a general program and could be used to display X3D with text in any language, eventually this would mean we have all UTF-8 glyphs in a texture, and that uses a lot of memory.

A proper solution will be done, but not in the immediate future, it's some work :) I'll leave this ticket open.

In the meantime, if you need it badly, I could provide a custom view3dscene version that includes e.g. German special characters in the font by default. This is trivial, requires just writing the sample German characters and recompiling. Let me know if this is interesting to you.

[brutzman]

Once Castle Game Engine has this Internationalization (I18N) capability, it will be fun to add some more countries and flags: Poland, Ukraine, others.

• X3D Example Archives: X3D4AM, X3D for Advanced Modeling, Hello World Scenes
• https://www.web3d.org/x3d/content/examples/X3dForAdvancedModeling/HelloWorldScenes

[michaliskambi]

Don, thanks for bumping this issue, this got me thinking how to finally solve it, in an easy way (instead of waiting long time for the "on-demand" solution mentioned in my post from 2020, https://github.com/castle-engine/view3dscene/issues/30#issuecomment-630715046 ).

Done :)

I changed a bit the default CGE fonts, and now we support by default (in all our fonts) "common Unicode characters", 2000 characters listed on https://gist.github.com/ivandrofly/0fe20773bd712b303f78 , which includes a lot of common languages.

• This includes Polish and German.
• @eugeneloza can you check and confirm that Ukrainian characters are also there?
• The languages excluded by default remain Japanese, Chinese (languages that require a lot of specific characters, and thus a lot of memory), but most other languages are now just supported out-of-the-box.

This increases some font texture sizes (from 256x256 to 1024x1024 or 2048x2048)... but it's 2023, such texture sizes are nothing special for any GPU, including mobiles. And it's just one such texture for font. We will actually load multiple textures if someone will really use X3D Text node with multiple font types (like https://github.com/castle-engine/demo-models/blob/master/text/text.x3dv , extreme example that will load 16x font textures + 1x font texture for UI; but almost no real application will use all these fonts).

Here's a screenshot from view3dscene doing test https://www.web3d.org/x3d/content/examples/X3dForAdvancedModeling/HelloWorldScenes/HelloGermanyIndex.html :

As usual, please give Jenkins a few hours and the view3dscene "snapshot" version from https://castle-engine.io/view3dscene.php will contain this fix. You can observe this page: https://github.com/castle-engine/castle-engine/compare/snapshot...master . When it will no longer contain the commit with message "Change default fonts to include all common Unicode characters..." then the given fix will be included in view3dscene snapshot download.

Dobrej zabawy z X3D! :)

[eugeneloza]

@eugeneloza can you check and confirm that Ukrainian characters are also there?

Unfortunately it seems like these letters are not available:

Є 0404, є 0454
І 0406, і 0456
Ї 0407, ї 0457

[michaliskambi]

Unfortunately it seems like these letters are not available:

Added then, thank you! :)

[michaliskambi]

Note: I know view3dscene builds with this enhancement are still not ready, as https://github.com/castle-engine/castle-engine/compare/snapshot...master shows. I'm working on it, Jenkins has trouble compiling all these fonts for all examples :)

Just wait for it, I monitor Jenkins and talk to it :) Once the commits will disappear from https://github.com/castle-engine/castle-engine/compare/snapshot...master then will be available in view3dscene "snapshot" version for sure. Give Jenkins a few more hours from this post.

[michaliskambi]

Note: I know view3dscene builds with this enhancement are still not ready, as https://github.com/castle-engine/castle-engine/compare/snapshot...master shows. I'm working on it, Jenkins has trouble compiling all these fonts for all examples :)

It took a bit more talking with Jenkins this time, but now the snapshots on https://castle-engine.io/view3dscene.php contain this improvement. Open "Hello World" with German letters in latest view3dscene snapshot and see :)

[brutzman]

Have added some new Hello World scenes for Poland and Ukraine.

Please check whether creator credits are OK, I could not have made these exemplars without each of your help.

• https://www.web3d.org/x3d/content/examples/X3dForAdvancedModeling/HelloWorldScenes/HelloPolandIndex.html
• https://www.web3d.org/x3d/content/examples/X3dForAdvancedModeling/HelloWorldScenes/HelloUkraineIndex.html

Plus there is a new feature in the pretty-print HTML documentation, hovering over an image url shows a preview on the right. This is helpful for quickly comparing screen captures.

All corrections and improvements welcome, thanks.

[michaliskambi]

@brutzman Thanks! But the Polish test on https://www.web3d.org/x3d/content/examples/X3dForAdvancedModeling/HelloWorldScenes/HelloPolandIndex.html is not testing Polish characters, as the 2 strings

Witaj Polsko!

and

Dobrej zabawy z X3D!

happen to not use any special Polish characters :) As you can see, they just use regular ASCII English characters. You don't need any special support in any browser to display them, beyond regular English characters. At it happens, most words in Polish are like that, i.e. they don't use special Polish characters, the special chars occur rarely in Polish.

If you want some examples with Polish characters, here are some texts that deliberately use special Polish chars:

Witaj Polsko! Ćma wącha źrebię.  # translation: "Hello Poland! The moth smells a pony."

Dobrej zabawy z X3D! Żółty żart. # translation: "Have fun with X3D! Yellow joke."

[brutzman]

Thanks, I think... While I like smelling a yellow pony covered with moths as much as anyone (cough cough), perhaps some other phrases might pertain?

"Hello World" programs usually try to be simple, that is why you see similar patterns between our scenes in this collection. Keeping them comparable and also interesting is helpful. Perhaps we can add another phrase with special characters, some Polish wisdom that is fun and motivating? Thanks in advance for all inspiration...

[michaliskambi]

Thanks, I think... While I like smelling a yellow pony covered with moths as much as anyone (cough cough), perhaps some other phrases might pertain?

Heh. Indeed my "test phrases" were a bit weird :)

In Polish, our "special characters" occur rarely, most Polish words can be done with pure ASCII (English) letters, so it is indeed a bit hard to construct some sensible sentence that also uses a lot of Polish special chars. Hm, how about

Żyć nie umierać!

Translation: literally "to live and not to die", it is a saying that's basically an encouragement to have fun and enjoy life or some opportunity or some gift you got.

Dzień dobry!

Translation: good morning! But this has just 1 Polish special char.

Gdyby kózka nie skakała, to by nóżki nie złamała.

A saying, still in wide use esp. to children, it means "if you wouldn't act so crazily, you wouldn't hurt yourself".

[michaliskambi]

I just realized that Polish characters have been actually missing from "common Unicode characters". We didn't render their glyphs out-of-the-box (despite me claiming we should do it, above, and even giving above Polish text samples...).

I was so sure that "common Unicode characters" includes Polish characters that I didn't check it. Shame on me. Added Polish characters now. Now Polish characters (in X3D in view3dscene, but also everywhere else in Castle Game Engine) are correctly rendered "out of the box".

As always, the fix will be part of view3dscene "snapshot" version when https://github.com/castle-engine/castle-engine/compare/snapshot...master will no longer show the relevant commit (titled "Add Polish characters").