[rcore] Draft: IME support

[daipom]

This is a fix for issue #1945. We are mainly assuming Japanese input, but this feature can be used for other languages using IME as well.

We are implementing IME support features in GLFW now:

• https://github.com/glfw/glfw/pull/2130

This fix adds the corresponding APIs to raylib.

It may take a while to be merged in GLFW, so we have created this PR as a draft ahead of time.

APIs to be added

• void SetPreeditCallback(PreeditCallback callback)
  • PreeditCallback: void (*PreeditCallback)(int preeditLength, int *preeditString, int blockCount, int *blockSizes, int focusedBlock, int caret)
  • Preedit version of CharCallback.
  • An app gets preedit info (preedit text, blocks, and caret position) by this callback and draws preedit.
  • The reason why we have an app register the callback directly, which is a different from CharCallback, is that this is not like CharCallback, where the input is resolved for each character, but rather requires managing the entire current preedit info at all times. Therefore, it is inefficient for an app to ask for preedit info every frame, so we use a callback to notify the change of preedit info.
• void SetPreeditCursorRectangle(int x, int y, int w, int h)
  • Set the area of the preedit cursor. This is used to decide the candidate window position that displays a list of conversion candidates. (For example, converting hiragana into kanji in Japanese).
  • In general, the position of this window should be managed by an app side by using this API.
• void GetPreeditCursorRectangle(int *x, int *y, int *w, int *h)
  • The getter version of SetPreeditCursorRectangle.
• void ResetPreedit(void)
  • Clear the current preedit.
• bool IsImeOn(void)
  • Get if the IME is ON.
• void SetImeStatus(bool on)
  • Set IME ON/OFF.

Feature to take candidate list (Win32 ONLY)

For a detailed explanation, please see below.

• void SetPreeditCandidateCallback(PreeditCandidateCallback callback)
  • PreeditCandidateCallback: void (*PreeditCandidateCallback)(int candidatesCount, int selectedIndex, int pageStart, int pageSize)
  • The app takes the current candidate status by this callback.
  • To get the candidate text for each index, use GetPreeditCandidate.
• int *GetPreeditCandidate(int index, int *textCount)
  • Get the current candiadate text of the specified index.

Specifications for each platform

Details are in the GLFW PR https://github.com/glfw/glfw/pull/2130.

Win32

• All APIs work fine.

macOS

• All APIs work fine.

X11

• Use over-the-spot style, so an app side doesn't need to draw preedit.
  • There are 2 main styles: on-the-spot and over-the-spot.
  • In on-the-spot style, an app side draws preedit, and this allows suitable drawing for each app. The other platforms use this style.
  • In over-the-spot style, the IME side draws preedit. This reduces what an app has to do, but it does not allow flexible display.
  • On X11, the behavior of on-the-spot style is not stable, so we use over-the-spot style.
• Only SetPreeditWindowPosition and GetPreeditWindowPosition work, and other APIs don't work.
• This is OK for this style because the IME side manages the drawing of preedit, so all an app side needs to do is managing the candidate window position.

Wayland

• Currently, we support the minimum required APIs: SetPreeditCallback, SetPreeditWindowPosition, and GetPreeditWindowPosition.

Sample Application

We are implementing the sample application RaylibIMEInputSampleApp.

• https://github.com/clear-code/RaylibIMEInputSampleApp

We can check all newly added APIs in this application.

Known issues

• X11: Some keys (arrow, Enter, BackSpace, ...) are passed to an app during preediting.
  • This will be fixed by https://github.com/glfw/glfw/pull/1972

Additional planned fixes

Since the fix of GLFW is large, we will submit separate PRs to GLFW for the following features. For this reason, we have also separated the fixes for raylib.

• Support soft fullscreen mode for IME features.
  • The candidate window doesn't be displayed correctly with fullscreen mode.
    • The default fullscreen is exclusive, and this causes the problem for IME displaying the candidate window.
  • We already have this fix for GLFW.
    • https://github.com/clear-code/glfw/tree/im-support-fullscreen
  • The corresponding fix of raylib is this.
    • https://github.com/clear-code/raylib/tree/better-ime-support-fullscreen
    • This will add FLAG_SOFT_FULLSCREEN config flag to make the fullscreen not exclusive.

Feature to take candidate list (Win32 ONLY)

Usually, the IME displays the candidate window, so the app side doesn't need to get canidate list.

However, sometimes, we need to display it on the app side. For example, it could be for displaying it correctly on exclusive fullscreen, or for using a custom design.

This is especially common in games for Windows.

This feature is OFF by default. We can use this by setting the FLAG_MANAGE_PREEDIT_CANDIDATE config flag.

Currently, this feature is available only on Win32, since this feature is especially common on Windows. We'd like to make it available on every platform if possible, but it may be difficult because of platform specific issues.

This feature is also involved in the issue with fullscreen and IME. If fullscreen is strongly exclusive, then the candidate window of IME can't be displayed correctly. This problem is especially serious in Windows, so this feature is one workaround for fullscreen on Windows.

We also have another fix that reduces fullscreen exclusivity for the IME, but the fix of GLFW for Windows is tricky... (Please see Additional planned fixes)

We can check this behavior in the sample application with MANAGE_PREEDIT_CANDIDATE cmake option.

• https://github.com/clear-code/RaylibIMEInputSampleApp/wiki/CMake-Build-Options#manage_preedit_candidate

[daipom]

It has not yet been merged into GLFW, so the fails of the tests are as expected.

Please see the wiki of RaylibIMEInputSampleApp about how to check this feature.

• https://github.com/clear-code/RaylibIMEInputSampleApp/wiki

[raysan5]

@daipom Thank you very much for all the hard work on this new feature! I couldn't review it carefully yet, probably some function could be renamed but the example looks fantastic! Thanks! 😄

[daipom]

Thanks! If you have any questions about this feature or see something to be fixed, please let me know!

[daipom]

I have just rebased this branch to the latest master.

[daipom]

@raysan5

In connection with #2814, many characters can be entered at one time when using an IME. So the current MAX_CHAR_PRESSED_QUEUE value is small for input using IME.

https://github.com/raysan5/raylib/blob/50a716c0d91f90e73f1fe2b910b3f7426a0d1360/src/rcore.c#L335-L337

It would be great if this could be made larger, such as 64, or if it could be changed on the application side. What would you think about this?

[daipom]

I have fixed some APIs: SetPreeditCursorRectangle, GetPreeditCursorRectangle.

[daipom]

I rebased this branch to the latest master and added the feature to take candidate list (Win32 ONLY). I have updated the initial comment on this PR, so please see it about this feature.

If you would like this feature to be separated into another PR, please let me know. As for this feature, it is summarized in the last commit and can be easily removed.

[raysan5]

@daipom Hi! It's been quite long since this amazing improvement was proposed, what is the current state? Is there some workaround to allow this feature to be added in raylib?

[Peter0x44]

This will definitely require some rebasing and modifications now the platform split is merged, to move the relevant code to rcore_desktop.c

[DreepyYunky]

I think It'll be better to change SetImeStatus() to TurnIMEOn() or InitIME().

[raysan5]

I just saw there was some update on this PR. I'm keeping it open for a bit longer, it seems lately GLFW is receiving several updates...

[ashie]

Hi, I'm a collaborator of this work. I've rebased this branch on top of current master (1fad8277a37b366841eff1497ca07a7042d4a014). The previous one is backed up as https://github.com/clear-code/raylib/tree/4.2-2022-12-14%2Bbetter-ime-support Since bundled GLFW isn't support IME yet, CI will fail as well as before.

I just saw there was some update on this PR. I'm keeping it open for a bit longer, it seems lately GLFW is receiving several updates...

IME support of GLFW isn't merged yet, it's planned as a part of v3.5: https://github.com/glfw/glfw/milestone/25 So we need to wait next minor update of GLFW (current is v3.4 released 2 weeks ago).

[ashie]

A commit is added automatically by CI: https://github.com/raysan5/raylib/pull/2809/commits/ccf2ddf1f88807d534a1833a3b548a7d73ea4b66

I'm not sure we can leave it as is, or we should do something more or not.