wip: normalize translation strings to NFC not NFKC

[matejcik]

[github-actions[bot]]

en main(all)

model	device_test	click_test	persistence_test
T2T1	test(all) main(all)	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)	test(all) main(all)

Translations

cs main(all)

model	device_test	click_test
T2T1	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)

de main(all)

model	device_test	click_test
T2T1	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)

es main(all)

model	device_test	click_test
T2T1	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)

fr main(all)

model	device_test	click_test
T2T1	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)

pt main(all)

model	device_test	click_test
T2T1	test(all) main(all)	test(all) main(all)
T3B1	test(all) main(all)	test(all) main(all)
T3T1	test(all) main(all)	test(all) main(all)
T3W1	test(all) main(all)	test(all) main(all)

Latest CI run: 16832989729

[bieleluk]

In pt test, the problematic character in string: is unsupported

[matejcik]

In pt test, the problematic character in string:

we need to generate font data for it and include it in the font jsons. i don't know how that is done, presumably some script in the tools directory is responsible for generating the special characters. there used to be data files with lists of special characters per language that should be generated

[bieleluk]

The rendered "˚" character is strangely rendered in the caesar layout: but also other layouts: WDYT @Hannsek , @obrusvit ?

[matejcik]

does the caesar font even contain that character? iirc it's a custom font. the square looks like a replacement character.

delizia looks right to me? https://en.wikipedia.org/wiki/Numero_sign#:~:text=The%20numero%20sign%2C%20either%20as,sometimes%20not%20used%20in%20abbreviations.

[bieleluk]

In the delizia case, I thought the horizontal bar should have been there as well, but according to Wikipedia, it actually doesn't have to be. Regarding the caesar case, the numero character is a special character, so it is not among the basic glyphs. I injected the unknown character into the translation on purpose, and it looks different. I guess the critical spot is the write_foreign_json function in the gen_font.py script

[bieleluk]

Rebasing due to conflicts with the base branch

[matejcik]

Regarding the caesar case, the numero character is a special character, so it is not among the basic glyphs. I injected the unknown character into the translation on purpose, and it looks different.

"unknown character" and "missing font glyph" are two different things. missing glyph rendering depends on the font engine: on Trezor we use the same picture as "unknown character" but on desktop you'll typically get a different picture, e.g., an empty square

i think that's what is going on here: you are asking the font engine to render you a "numero character" and getting a square

[bieleluk]

You are right, @matejcik, some fonts do not support 'º' (U+00BA):

PixelOperator-Bold.ttf        | ❌ does NOT support
PixelOperator-Regular.ttf     | ❌ does NOT support
PixelOperatorMono-Regular.ttf | ❌ does NOT support
Roboto-Bold.ttf               | ✅ supports
Roboto-Regular.ttf            | ✅ supports
RobotoMono-Light.ttf          | ✅ supports
RobotoMono-Medium.ttf         | ✅ supports
RobotoMono-Regular.ttf        | ✅ supports
TTHoves-Bold.otf              | ✅ supports
TTHoves-DemiBold.otf          | ✅ supports
TTHoves-Medium.otf            | ✅ supports
TTHoves-Regular.otf           | ✅ supports
TTSatoshi-Bold.otf            | ✅ supports
TTSatoshi-DemiBold.otf        | ✅ supports
TTSatoshi-ExtraLight.otf      | ✅ supports
TTSatoshi-Medium.otf          | ✅ supports
TTSatoshi-Regular.otf         | ✅ supports
Unifont-Bold.otf              | ❌ does NOT support
Unifont-Regular.otf           | ✅ supports

In those fonts, I used a mapping to plain 'o' to avoid rendering an empty square: https://github.com/trezor/trezor-firmware/pull/5106/commits/6eb89ac494551d28137d222d2a906d4cf564fb41

[bieleluk]

Rebasing due to a conflict with a base branch ...