test(developer): Regression Tests for Keyman Developer Alpha

[MakaraSok]

Acceptance testing of the latest Keyman Developer alpha before going into Beta.

Keyman Developer Acceptance Test Procedures

1. These test procedures are to be run before moving from alpha to beta, or beta to stable, or before PRs are merged into stable branches.
2. Copy these checklists of tests into a new issue (for tier transitions) or a comment on the PR.

User Testing

Gather Assets for Testing

• Windows 10 (x64) - VM or physical okay
• Windows 11 (x64) - VM or physical okay (if prefer for other suites)
• Keyman Developer build from 16.0.41

Setup Steps

• Uninstall previous versions of Keyman for Windows and Keyman Developer on the system
• Restart system, install updates
• Clean up any broken installation artifacts (ideally, roll back to clean snapshot in VM):
  • %ProgramFiles(x86)%\Keyman or %ProgramFiles%\Keyman
  • %CommonProgramFiles(x86)%\Keyman or %CommonProgramFiles%\Keyman
  • %ProgramData%\Keyman
  • %AppData%\Keyman
  • %LocalAppData%\Keyman
  • HKCU\Software\Keyman
  • HKLM\Software\Wow6432Node\Keyman
  • Manually remove any remaining Keyman input methods from Windows Languages

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

GROUP_WIN10 - to be tested on Windows 10 (x64)

GROUP_WIN11 - to be tested on Windows 11 (x64)

• TEST_STARTS: Verify that Keyman Developer starts
• TEST_COMPILES: Open a .kpj project, load keyboards and verify that you can compile them

What's New: test new functionality

SUITE_DEBUGGER: Test new core-based debugger (WIN10)

Setup: open a reasonably complex keyboard project (eg. Amharic and Cameroon keyboards), load the .kmn, and press F7 to compile.

• TEST_DEBUGGER_STARTS: Start debugging by pressing F5. Verify that the debugger starts and you can type with the keyboard in the debug window. Look out for any unexpected behaviour or visual oddities.
• TEST_DEBUGGER_SINGLE_STEP: Press the Single Step button (>>) in the Debugger toolbar. Type a key in the debugger window. Use the debugger toolbar (>) button to step through the rules and verify that the matched rules are correct. Examine the Debugger State windows (Elements, Call stack, etc) to verify that the content presented at each point makes sense.
• TEST_DEBUGGER_PAUSE: Press the Pause button (||) in the Debugger toolbar. Verify that no input is possible in the debugger window.
• TEST_DEBUGGER_CHANGES: Make a change to the source file and verify that the debugger works correctly after recompiling.
• TEST_DEBUGGER_PLATFORM: Create some platform-specific rules with platform() statement, and use the Debugger State window 'Platform' to select platform variants, and verify that the debugger selects the appropriate rules in testing.

SUITE_SERVER: Test new Keyman Developer Server (WIN10)

• TEST_SERVER_STARTS: Start Keyman Developer. Verify that Server starts by examining the tool tray area, and launching http://localhost:8008 in your browser.

• TEST_SERVER_STOPS: Start Keyman Developer. Check that Tools/Options/Server/Leave Server running after closing IDE is off. Exit Keyman Developer. Verify that Server has shut down by attempting to visit http://localhost:8008 in your browser -- it should fail to load.

• TEST_SERVER_KEEP_ALIVE: Start Keyman Developer. Check that Tools/Options/Server/Leave Server running after closing IDE is on. Exit Keyman Developer. Verify that Server has is still running by visiting http://localhost:8008 in your browser.

• TEST_SERVER_KEYBOARD: Make sure that a keyboard debug session works.

  1. In Keyman Developer, load a keyboard project, open a keyboard, and start testing on web.
  2. Open http://localhost:8008 in your browser, and select the keyboard from the Keyboards drop down
  3. Verify that the keyboard is working in the web debug view.

• TEST_SERVER_KEYBOARD_LIVE_CHANGE: Make sure that changes to the keyboard are dynamically updated.

  1. Follow TEST_SERVER_KEYBOARD steps to start debugging the keyboard.
  2. Make a visual change to the on screen keyboard / touch keyboard view in the keyboard and recompile it (F7</kbd).
  3. Switch back to the browser and verify that the change is visible in the keyboard.

SUITE_FLICK_AND_MULTITAP (WIN10)

Here, we aim to test that touch layout designer functionality available in earlier versions still works in the new version.

• TEST_ADD_KEY: Add a key to an existing row of the keyboard (with the green 'wedge').
• TEST_ADD_ROW: Add a new row to an existing layer on the keyboard (with the green 'wedge').
• TEST_ADD_LAYER: Add a new layer to a keyboard (Layer/Add button)
• TEST_DEL_KEY: Delete a key from the keybaord (Red x button on the selected key)
• TEST_DEL_ROW: Delete a row from the keyboard by deleting all keys in the row.
• TEST_DEL_LAYER: Delete a layer from the keyboard (Layer/Del button).
• TEST_ADD_PLATFORM: Add either tablet or phone platform.
• TEST_DEL_PLATFORM: Delete either tablet or phone platform from the keyboard.
• TEST_TEMPLATE: Import from On Screen, then use the Template button to select the Basic layout, and verify that this generates a 'basic' layout -- mobile-optimised, fewer keys than a desktop layout.
• TEST_SWITCH_LAYER: Select an alternative layer from the Layer menu. Then double-click on the Shift key to switch layers another way.
• TEST_PRESENTATIONS: Switch between the landscape/portrait orientation and verify correct.

Next, we'll want to test the key cap editing behaviour.

• TEST_EDIT_TEXT: Select a key and change its key cap value in the Text field.
• TEST_EDIT_UNICODE: Select a key and change its key cap value in the Unicode field.
• TEST_EDIT_ID: Select a key and change its ID.
• TEST_EDIT_PAD: Select a key and change the padding to a value such as 30. Verify that the padding left of the key grows.
• TEST_EDIT_WIDTH: Select a key and change its width to the value 200. Verify that the key is twice as wide as normal.

Edit gesture data.

• TEST_ADD_LONGPRESS: For a selected base key, add a longpress by clicking the [+] button in the Longpress area. Verify that the longpress icon appears on the base key.
• TEST_DEL_LONGPRESS: For a selected base key, delete a longpress key by clicking the red (x) button on the longpress key.
• TEST_ADD_FLICK: For a selected base key, add a flick by clicking any [+] button in the Flick area. Verify that the coresponding Flick icon appears on the base key.
• TEST_DEL_FLICK: For a selected base key, delete a flick key by clicking the red (x) button on the flick key.
• TEST_ADD_MULTITAP: For a selected base key, add a multitap by clicking the [+] button in the Multitap area. Verify that the multitap icon appears on the base key.
• TEST_DEL_MULTITAP: For a selected base key, delete a multitap key by clicking the red (x) button on the multitap key.

Drag and drop a character from Character maps onto a key.

• TEST_CHAR_DROP: Drag a character from character map and drop it onto a key. The new character should replace the character on the key.
• TEST_CHAR_DROP_ADD: Drag a character from character map and drop it onto a key while holding Shift. The new character should be added to the key.
• TEST_CHAR_DROP_TEXT: Drag a character from character map and drop it onto the Text field. The new character should replace the character on the key.
• TEST_CHAR_DROP_TEXT_ADD: Drag a character from character map and drop it onto the Text field while holding Shift. The new character should be added to the key.

[keymanapp-test-bot[bot]]

User Test Results

Test specification and instructions

✅ SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

4 tests in 2 groups PASSED

• ✅ GROUP_WIN10: - to be tested on Windows 10 (x64)

  2 tests PASSED

  • ✅ TEST_STARTS (PASSED): Verified the Keyman Developer 16.0.42 build and it successfully opens.
  • ✅ TEST_COMPILES (PASSED): Able to compile the loaded keyboards successfully using Keyman Developer.

• ✅ GROUP_WIN11: - to be tested on Windows 11 (x64)

  2 tests PASSED

  • ✅ TEST_STARTS (PASSED): Verified the Keyman Developer 16.0.42 build and it successfully opens.
  • ✅ TEST_COMPILES (PASSED): Able to compile the loaded keyboards successfully using Keyman Developer.

✅ SUITE_DEBUGGER: Test new core-based debugger (WIN10)

5 tests in 1 groups PASSED

• ✅ TEST_DEBUGGER_STARTS (PASSED): Verified that the debugger started successfully and I am able to type with the keyboard in the debug window without having any problem.
• ✅ TEST_DEBUGGER_SINGLE_STEP (PASSED): Tested this as per the instructions and examined the Debugger tabs like Windows, Elements, Call Stack etc., and verified that the content presented at each point makes sense.
• ✅ TEST_DEBUGGER_PAUSE (PASSED): Verified that no input is possible after I clicked the Pause button.
• ✅ TEST_DEBUGGER_CHANGES (PASSED): Verified the debugger works correctly after the recompiling a source file.
• ✅ TEST_DEBUGGER_PLATFORM (PASSED): Tested this as per Makara's instructions in Windows 10 OS and the I was able to create some specific rules and verified that the debugger selects the rules in appropriate testing.

✅ SUITE_SERVER: Test new Keyman Developer Server (WIN10)

5 tests in 1 groups PASSED

• ✅ TEST_SERVER_STARTS (PASSED): Verified the Server starts by examining the tool tray area, and launching http://localhost:8008 in my browser.
• ✅ TEST_SERVER_STOPS (PASSED): After the Server was shut down, the above mentioned browser fails to load.
• ✅ TEST_SERVER_KEEP_ALIVE (PASSED): Verified the Server has still running by visiting http://localhost:8008 in my browser after 'Leave Server running after closing IDE ' is ON.
• ✅ TEST_SERVER_KEYBOARD (PASSED): Verified the selected keyboard is working fine in the web debug view.
• ✅ TEST_SERVER_KEYBOARD_LIVE_CHANGE (PASSED): Verified the changes to the keyboard are dynamically updated in the browser.

✅ SUITE_FLICK_AND_MULTITAP: (WIN10)

26 tests in 1 groups PASSED

• ✅ TEST_ADD_KEY (PASSED): Able to add Key in an existing row.
• ✅ TEST_ADD_ROW (PASSED): Able to add a new row to an existing layer on the keyboard.
• ✅ TEST_ADD_LAYER (PASSED): Able to add a new layer to a keyboard using the Add button.
• ✅ TEST_DEL_KEY (PASSED): Able to delete a key from the keyboard.
• ✅ TEST_DEL_ROW (PASSED): Able to delete a row by deleting all keys in the row.
• ✅ TEST_DEL_LAYER (PASSED): Able to delete a layer from the keyboard.
• ✅ TEST_ADD_PLATFORM (PASSED): Able to add a platform (tablet / phone) by using the Add button.
• ✅ TEST_DEL_PLATFORM (PASSED): Able to delete a platform (tablet / phone) bu using the Delete button.
• ✅ TEST_TEMPLATE (PASSED): Tested this as per the instructions and it is working as expected.
• ✅ TEST_SWITCH_LAYER (PASSED): Able to switch between the two layers. (eg., from Shift layer to other layer)
• ✅ TEST_PRESENTATIONS (PASSED): Able to switch between the landscape/portrait orientation.
• ✅ TEST_EDIT_TEXT (PASSED): Able to change the key cap value in the Text field.
• ✅ TEST_EDIT_UNICODE (PASSED): Able to change the key cap value in the Unicode field.
• ✅ TEST_EDIT_ID (PASSED): Able to change the Key's ID.
• ✅ TEST_EDIT_PAD (PASSED): Tested this as per the instructions and it is working as expected.
• ✅ TEST_EDIT_WIDTH (PASSED): Verified the key is twice as wide as normal.
• ✅ TEST_ADD_LONGPRESS (PASSED): Verified the Long Press icon that appears on the base key.
• ✅ TEST_DEL_LONGPRESS (PASSED): Able to delete the Longpress key by clicking the red (x) button on the longpress key.
• ✅ TEST_ADD_FLICK (PASSED): Verified the corresponding Flick icon appears on the base key.
• ✅ TEST_DEL_FLICK (PASSED): Able to delete the Flick key by clicking the red (x) button on the flick key.
• ✅ TEST_ADD_MULTITAP (PASSED): Able to add the Multitap to a base key.
• ✅ TEST_DEL_MULTITAP (PASSED): Able to delete the Multitap icon on the base key.
• ✅ TEST_CHAR_DROP (PASSED): Able to replace the character on the key.
• ✅ TEST_CHAR_DROP_ADD (PASSED): Able to add a new character to the existing character key.
• ✅ TEST_CHAR_DROP_TEXT (PASSED): Able to replace a key using Text field.
• ✅ TEST_CHAR_DROP_TEXT_ADD (PASSED): Able to add a new character with an existing character key using Text field.

[bharanidharanj]

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

GROUP_WIN10: - to be tested on Windows 10 (x64)

• TEST_STARTS (PASSED): Verified the Keyman Developer 16.0.42 build and it successfully opens.
• TEST_COMPILES (PASSED): Able to compile the loaded keyboards successfully using Keyman Developer.

[bharanidharanj]

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

SUITE_DEBUGGER: Test new core-based debugger (WIN10)

• TEST_DEBUGGER_STARTS (PASSED): Verified that the debugger started successfully and I am able to type with the keyboard in the debug window without having any problem.
• TEST_DEBUGGER_SINGLE_STEP (PASSED): Tested this as per the instructions and examined the Debugger tabs like Windows, Elements, Call Stack etc., and verified that the content presented at each point makes sense.
• TEST_DEBUGGER_PAUSE (PASSED): Verified that no input is possible after I clicked the Pause button.
• TEST_DEBUGGER_CHANGES (PASSED): Verified the debugger works correctly after the recompiling a source file.

[bharanidharanj]

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

SUITE_SERVER: Test new Keyman Developer Server (WIN10)

• TEST_SERVER_STARTS (PASSED): Verified the Server starts by examining the tool tray area, and launching http://localhost:8008 in my browser.
• TEST_SERVER_STOPS (PASSED): After the Server was shut down, the above mentioned browser fails to load.
• TEST_SERVER_KEEP_ALIVE (PASSED): Verified the Server has still running by visiting http://localhost:8008 in my browser after 'Leave Server running after closing IDE ' is ON.
• TEST_SERVER_KEYBOARD (PASSED): Verified the selected keyboard is working fine in the web debug view.
• TEST_SERVER_KEYBOARD_LIVE_CHANGE (PASSED): Verified the changes to the keyboard are dynamically updated in the browser.

[bharanidharanj]

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

SUITE_FLICK_AND_MULTITAP: (WIN10)

• TEST_ADD_KEY (PASSED): Able to add Key in an existing row.
• TEST_ADD_ROW (PASSED): Able to add a new row to an existing layer on the keyboard.
• TEST_ADD_LAYER (PASSED): Able to add a new layer to a keyboard using the Add button.
• TEST_DEL_KEY (PASSED): Able to delete a key from the keyboard.
• TEST_DEL_ROW (PASSED): Able to delete a row by deleting all keys in the row.
• TEST_DEL_LAYER (PASSED): Able to delete a layer from the keyboard.
• TEST_ADD_PLATFORM (PASSED): Able to add a platform (tablet / phone) by using the Add button.
• TEST_DEL_PLATFORM (PASSED): Able to delete a platform (tablet / phone) bu using the Delete button.
• TEST_TEMPLATE (PASSED): Tested this as per the instructions and it is working as expected.
• TEST_SWITCH_LAYER (PASSED): Able to switch between the two layers. (eg., from Shift layer to other layer)
• TEST_PRESENTATIONS (PASSED): Able to switch between the landscape/portrait orientation.
• TEST_EDIT_TEXT (PASSED): Able to change the key cap value in the Text field.
• TEST_EDIT_UNICODE (PASSED): Able to change the key cap value in the Unicode field.
• TEST_EDIT_ID (PASSED): Able to change the Key's ID.
• TEST_EDIT_PAD (PASSED): Tested this as per the instructions and it is working as expected.
• TEST_EDIT_WIDTH (PASSED): Verified the key is twice as wide as normal.
• TEST_ADD_LONGPRESS (PASSED): Verified the Long Press icon that appears on the base key.
• TEST_DEL_LONGPRESS (PASSED): Able to delete the Longpress key by clicking the red (x) button on the longpress key.
• TEST_ADD_FLICK (PASSED): Verified the corresponding Flick icon appears on the base key.
• TEST_DEL_FLICK (PASSED): Able to delete the Flick key by clicking the red (x) button on the flick key.
• TEST_ADD_MULTITAP (PASSED): Able to add the Multitap to a base key.
• TEST_DEL_MULTITAP (PASSED): Able to delete the Multitap icon on the base key.
• TEST_CHAR_DROP (PASSED): Able to replace the character on the key.
• TEST_CHAR_DROP_ADD (PASSED): Able to add a new character to the existing character key.
• TEST_CHAR_DROP_TEXT (PASSED): Able to replace a key using Text field.
• TEST_CHAR_DROP_TEXT_ADD (PASSED): Able to add a new character with an existing character key using Text field.

[bharanidharanj]

Test Results

SUITE_DEVELOPER_BASELINE: Keyman Developer baseline tests

GROUP_WIN11: - to be tested on Windows 11 (x64)

• TEST_STARTS (PASSED): Verified the Keyman Developer 16.0.42 build and it successfully opens.
• TEST_COMPILES (PASSED): Able to compile the loaded keyboards successfully using Keyman Developer.

[bharanidharanj]

SUITE_DEBUGGER: Test new core-based debugger (WIN10)

• TEST_DEBUGGER_PLATFORM (PASSED): Tested this as per Makara's instructions in Windows 10 OS and the I was able to create some specific rules and verified that the debugger selects the rules in appropriate testing.

[mcdurdin]

LGTM