fix(developer): correct use of u16chr when second parameter could be null

[markcsinclair]

Fix the use of u16chr() with *q==0 in u16tok() bug (and anywhere else it is similarly misused) - see issue #11814

Tests for u16chr() and u16tok() (both variants) are included in gtest-kmxu16-test.cpp

Partially fixes: #11814 (another PR is needed for the first bug)

@keymanapp-test-bot skip

[keymanapp-test-bot[bot]]

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• Keyboards
  • Test Keyboards

[mcdurdin]

I independently encountered the u16tok bug. I have pushed my branch so you can compare: #11910.

[markcsinclair]

I think the tests for u16chr and u16tok (both variants) are complete, so the next task is to identify and correct any additional incorrect uses of u16chr, where second parameter could be null.

Possibilities include:

• ProcessEthnologueStore() (see here)
• LineTokenType() (two places, see here)

[markcsinclair]

Ready for review, although waiting for PR #11910 to be merged into master first.

[markcsinclair]

PR #11910 has been merged into master, and in turn into this PR, so it is ready to be reviewed.

Compared to PR #11910, this PR offers more extensive testing of u16chr() and the two variants of u16tok(), plus testing of two Compiler.cpp functions, ProcessEthnologueStore() and LineTokenType() which were (incorrectly) suspected of containing misuse of u16chr(). The testing on ProcessEthnologueStore() did identify a different bug (see #11955) however.

[markcsinclair]

Noting that the title and description suggest changes to the compiler code, not just the tests -- is there something missing?

This PR originally contained a fix to the compiler whilst in draft, but then you developed a fix in PR #11910. I merged that into this PR, adopting your fix and extending my tests (see my comments above). I will change the name of this PR to a reflect its final test-only content.

[mcdurdin]

This PR originally contained a fix to the compiler whilst in draft, but then you developed a fix in PR #11910. I merged that into this PR, adopting your fix and extending my tests (see my comments above). I will change the name of this PR to a reflect its final test-only content.

And ... I knew that and forgot. Sorry for the hassle!

[keyman-server]

Changes in this pull request will be available for download in Keyman version 18.0.80-alpha