PoC : a x230 user has keyboard issue being raw translated 2 instead of AT translated 2

[tlaurion]

This is PoC testing with user having inconsistent issues. Randomly properly detected on boot and not, just like for x230t under https://github.com/osresearch/heads/issues/958 which was never completely resolved.

First things first

• deactivate coreboot ps2 init and let linux do that is thought right.

[tlaurion]

Well, last time we attempted to fix this was on 4.14 kernel and coreboot 4.13 at https://github.com/linuxboot/heads/pull/1302#issuecomment-1412290374

The fix above doesn't expect to fix issue with x230t, but maybe will fix issue that wasn't there for user prior of flashing from PB (coreboot 4.13 and kernel 4.14) to latest. Let's still check that first.

[tlaurion]

Without coreboot ps2 init, user reports no joy. With i8042.notimeout: EDIT: not much more successful coldboot.

[tlaurion]

I have no clue what to do with that. We see that successful and failing behavior is linked with configuration of ps2 controller through i8042 which is not configuring the keyboard the same way.

Observations:

• on coldboot, user gets bad behavior from i8042 probing keyboard which results in input: AT Raw Set 2 keyboard under Qubes only.
• On reboot from either Heads or QubesOS (Or any OS) user gets good keyboard input: AT Translated Set 2 keyboard

Following logs have been stripped for comparison doing:

grep -Rn i8042 dom0_dmesg_fail.txt > dom0_dmesg_i8042_fail.txt
grep -Rn i8042 dom0_dmesg_success.txt > dom0_dmesg_i8042_success.txt
cat dom0_dmesg_i8042_success.txt| awk -F 'localhost' {'print $2'} > dom0_dmesg_i8042_success_stripped.txt
cat dom0_dmesg_i8042_success.txt| awk -F 'localhost' {'print $2'} > dom0_dmesg_i8042_success_stripped.txt

And then anything after dectection of keyboard has been removed to protect user's typed keystrokes related privacy/disclosure of irrelevant information. dom0_dmesg_i8042_success_stripped.txt dom0_dmesg_i8042_fail_stripped.txt

[tlaurion]

Let's analyse the debug logs above.

• In the "fail" debug output, the command byte that is read from the i8042 controller is 0x20, which means that the bit 6 is 0 and the translation mode is disabled. In the "success" debug output, the command byte that is read from the i8042 controller is 0x65, which means that the bit 6 is 1 and the translation mode is enabled. This is the main difference that affects the keyboard detection.
• In the "fail" debug output, the keyboard identification code that is returned by the keyboard after the 0xf2 command is 0xab 0x84, which means that the keyboard is using the AT Raw Set 2 protocol. In the"success" debug output, the keyboard identification code that is returned by the keyboard after the 0xf2 command is 0xab 0x54, which means that the keyboard is using the AT Translated Set 2 protocol. This is the result of the translation mode being enabled or disabled.
• In the "failed" debug output, the input device name that is created by the i8042 driver for the keyboard is AT Raw Set 2 keyboard, which means that the keyboard is using the raw protocol and not the standard input event interface. In the "success" debug output, the input device name that is created by the i8042 driver for the keyboard is AT Translated Set 2 keyboard, which means that the keyboard is using the translated protocol and the standard input event interface. This is the consequence of the keyboard identification code being different.

[tlaurion]

If we focus on the coreboot code for ps2 keyboard detection

PS/2 Keyboard Detection Issue on x230 with coreboot

Problem Description

The user reported that their PS/2 keyboard on the x230 laptop with coreboot is sometimes detected as a raw keyboard and sometimes as a translated keyboard. The user provided two debug outputs that show the difference in the keyboard detection. The user wanted to understand why this is happening and how to fix it.

Analysis and Findings

• The keyboard detection is affected by the translation mode of the i8042 controller, which is a hardware device that communicates with the PS/2 keyboard and mouse. The translation mode converts the raw scan codes of the keyboard to the standard input event codes of the system. The translation mode is controlled by the bit 6 of the command byte, which is a register that stores the configuration of the i8042 controller.
• The translation mode is enabled or disabled by the coreboot code that initializes the i8042 controller and the keyboard. The coreboot code is located in the src/drivers/i8042 directory of the coreboot repository. There are some files that are relevant to the keyboard detection, such as i8042.c, i8042.h, and ps2_keyboard.c. These files contain some macros and functions that define the timeout values and the error handling for the i8042 controller and the keyboard.
• The keyboard protocol is affected by the translation mode and not by the keyboard itself. The keyboard protocol determines how the keyboard sends and receives data to and from the host. The keyboard protocol can be either AT Raw Set 2 or AT Translated Set 2. The AT Raw Set 2 protocol sends raw scan codes to the host, while the AT Translated Set 2 protocol sends translated input event codes to the host. The keyboard protocol is identified by the 0xf2 command, which returns a two-byte identification code from the keyboard.
• The input device name is affected by the keyboard protocol and not by the input layer. The input device name is the name that the system assigns to the keyboard when it creates an input device for it. The input device name can be either AT Raw Set 2 keyboard or AT Translated Set 2 keyboard. The AT Raw Set 2 keyboard uses the raw protocol and not the standard input event interface, while the AT Translated Set 2 keyboard uses the translated protocol and the standard input event interface. The input device name is created by the i8042 driver, which is the driver that handles the communication between the i8042 controller and the keyboard.

Possible Solutions

Based on the analysis and findings, I suggest the following possible solutions to fix the keyboard detection issue by modifying the coreboot code:

• In the i8042.c file, there is a function called i8042_flush that flushes the output buffer of the i8042 controller by reading and discarding any data that is present. This function uses a loop that checks the status register of the controller and reads the data register if the output buffer is full. The loop has a timeout value of I8042_CTL_TIMEOUT milliseconds, which is defined as 250 in the i8042.h file. If the timeout is reached, the function returns an error code of -1. This may cause the i8042 driver to fail to communicate with the controller or the keyboard if the output buffer is not emptied within the timeout value. To fix this, increase the value of the I8042_CTL_TIMEOUT macro in the i8042.h file to a higher value, such as 500 or 1000 milliseconds. This may make the i8042 driver more tolerant to slow devices.
• In the ps2_keyboard.c file, there is a function called ps2_keyboard_init that initializes the keyboard by sending and receiving various commands and data. This function uses a loop that tries to send the 0xf2 command to the keyboard and read the identification code from the keyboard. The loop has a retry value of 3, which means that the function will try to send the command and read the code up to 3 times before giving up. If the retry value is exhausted, the function returns an error code of -1. This may cause the i8042 driver to fail to identify the keyboard if the keyboard does not respond to the command or send the code within the retry value. To fix this, increase the value of the retry value in the ps2_keyboard_init function to a higher value, such as 5 or 10. This may make the i8042 driver more persistent to identify the keyboard.
• In the i8042.c file, there is a function called i8042_init that initializes the i8042 controller and the keyboard. This function reads and writes the command byte of the controller to enable or disable the translation mode for the keyboard port. The function uses a macro called I8042_XLATE to set the bit 6 of the command byte to 1 or 0. The macro is defined as 0x40 in the i8042.h file. If the macro is 0x40, the translation mode is enabled. If the macro is 0x00, the translation mode is disabled. This may cause the keyboard to use the raw or the translated protocol depending on the macro value. To fix this, change the value of the I8042_XLATE macro in the i8042.h file to 0x40. This may make the i8042 driver always enable the translation mode for the keyboard port.

PoCs.

To fix the keyboard detection issue and make it always use the AT Translated Set 2 protocol

--- a/src/drivers/i8042/i8042.h
+++ b/src/drivers/i8042/i8042.h
@@ -23,7 +23,7 @@
 #define I8042_AUX_OBF		0x20	/* output buffer full */
 #define I8042_AUX_IRQ		12
 
-#define I8042_XLATE		0x00	/* translate scan code set 2 to 1 */
+#define I8042_XLATE		0x40	/* translate scan code set 2 to 1 */
 #define I8042_MUX_QUIRK		0x02	/* enable multiplexing quirk */
 
 #define I8042_KBD_IRQ		1

Only for x230:

--- a/src/drivers/i8042/i8042.c
+++ b/src/drivers/i8042/i8042.c
@@ -132,7 +132,11 @@ static int i8042_init(void)
 	/* Enable keyboard interface */
 	i8042_command(&param, I8042_CMD_CTL_WCTR);
 
+#if CONFIG_BOARD_LENOVO_X230
+	param |= I8042_XLATE;
+#else
 	param &= ~I8042_XLATE;
+#endif
 	i8042_command(&param, I8042_CMD_CTL_WCTR);
 
 	/* Flush any pending input. */

Sharing this on coreboot chat channel.

[tlaurion]

Note that the ps2 driver was added into coreboot 4.20....

[tlaurion]

Ok confirmed from user that pc80 under coreboot is at cause, but too much debug to give anything useful from heads dmesg which is filled with i8042 logs as of now. But keyboard is properly detected as can be seen

Coldboot

warmboot

[aless3]

I have thinkpad x230 with heads (not sure the version) and arch Linux as OS, I had this exact problem more or less a few months ago, I think it was a bug in systemd, I found the error appearing in versions from systemd 253.2+, later resolved (I don't know when exactly, about mid 254 or 255?) the problem appeared without touching heads, it was the same installation, but I upgraded arch. I noticed that bug was with systemd because "locking" the packages to before 2023/03/29 had no issues and of the few package upgrades of the day after systemd was the one causing it

I am not really able to use my laptop now, but I'd say first watch if using systemd and its version and try a downgrade (first to 252.x, but all of them should work up to 253.1)

[tlaurion]

I have thinkpad x230 with heads (not sure the version) and arch Linux as OS, I had this exact problem more or less a few months ago, I think it was a bug in systemd, I found the error appearing in versions from systemd 253.2+, later resolved (I don't know when exactly, about mid 254 or 255?) the problem appeared without touching heads, it was the same installation, but I upgraded arch. I noticed that bug was with systemd because "locking" the packages to before 2023/03/29 had no issues and of the few package upgrades of the day after systemd was the one causing it

I am not really able to use my laptop now, but I'd say first watch if using systemd and its version and try a downgrade (first to 252.x, but all of them should work up to 253.1)

Its a different thing if OS applies udev/systemd hacks on detected hardware IDs then coreboot not being able to detect the keyboard correctly in the sense of talking with it correctly/iitializing it correctly creating different behaviors across reboot (warm reset: not all components being resetted) and cold boot (cold power on) or even on reset.

I just pushed an ugly hack under 1a6fbfa for a user to test and report behavior from cold boot (which is source of issue) and reboot, giving info from the source (cbmem -1 giving ps2 probe/detection/errors) and Heads dmesg output.

This patch is augmenting timeout of pc80 code, removing exit codes on two of the functions (leaving debugging info for coreboot devels to take decision) and continuing instead of abording, since pc80 code following is actually the one responsible to report of how ps2 keyboard communicates through IOs.

This is definitely not production ready and a plain PoC on top of master, but the output should convince coreboot that there is an issue and that it should be resolved for x230t/x220t and now x230 users of coreboot+linux, where otherwise the keyboard is left in a weird state where workarounds need to be applied from OS (where otherwise seabios has workarounds that are applied automatically and hides the problem).

[tlaurion]

Next steps https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-1817528804

[tlaurion]

Another attempt to discuss over coreboot channel with collected evidence from users testing of above ugly patch: https://matrix.to/#/!EhaGFZyYcbyhdSgStq:matrix.org/$vgBbogOreZfS41d-yJInWqAUUgvUTvfJLzLlokfMUY8?via=matrix.org&via=libera.chat&via=sibnsk.net (self-tracking)

icon on coreboot suggests to try to adapt libpayload quirks to pc80 or rewrite something else since enabling ps2 keyboard under coreboot is adding delay on boot. Also suggests that the payload (linux) should do it properly. Unfortunately, I'm not sure I get the details properly. As I currently understand the flow of pc80 code, even if everything goes wrong, we could bypass all timeouts and tests (exit 0 removed in patch) and set AT 2 translated mode and be happy. Otherwise quirks attempted (i8042.notimeout, i8042.reset, i8042.translate=1:kbd were not helpful having coldboot/warm boot having AT Translated 2 consistently for some users. Is it EC firmware fault, keyboard SKU, things are unclear and I can't replicate).

[tlaurion]

Opened coreboot issue https://ticket.coreboot.org/issues/515

[tlaurion]

I would really prefer not going that path, but this is pseudo code generated by bing discussion that could be applied to i8042 and start discussion with Linux folks for linuxboot use cases.

Disclaimer: Not tested, might be wrong and would not want to maintain that forever as patch under heads but if works with coreboot ps2 keyboard deactivated, might be useful to prove validity of new option and would permit boards that need it to just enable it under correhoot->Linux COMMAND_LINE to initialize ps2 keyboard properly from the "payload" (firmware) by simply adding "i8042.forceat2" there.

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1004,6 +1007,16 @@ static int i8042_command(struct i8042_port *port, unsigned char *param, int comm
 		return retval;
 	}
 
+	if (i8042_forceat2 && port == &i8042_ports[I8042_KBD_PORT]) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (command)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	if (command & I8042_CMD_AUX_SEND) {
 		retval = i8042_wait_write();
 		if (retval)

[tlaurion]

An optimized first approach to patch the i8042 Linux kernel code is to modify the i8042_kbd_init function in the i8042.c file, which is responsible for initializing and probing the PS/2 keyboard. This approach adds a code block that sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This is similar to what you did in the pc80 keyboard.c file in the heads project. This approach also adds a new kernel parameter, i8042.forceat2, that enables this modification. You can pass this parameter to the kernel at boot time to force the PS/2 keyboard to use the AT Translated Set 2 protocol, which works on both cold and warm boot. Here is what the code for this approach looks like:

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1138,6 +1141,16 @@ static int __init i8042_kbd_init(void)
 		return -ENODEV;
 	}
 
+	if (i8042_forceat2) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (kbd init)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	if (i8042_dumbkbd) {
 		i8042_kbd_read_data = i8042_dumbkbd_read_data;
 		i8042_kbd_read_status = i8042_dumbkbd_read_status;

This code adds a conditional statement that checks if the i8042.forceat2 kernel parameter is enabled. If the condition is true, it sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This code is executed only once, after the keyboard is initialized and probed. This avoids sending the command and the argument unnecessarily every time the i8042_command function is called.

[tlaurion]

A second optimized approach to patch the i8042 Linux kernel code is to modify the i8042_kbd_get_id function in the i8042.c file, which is responsible for sending the probe command (0xF2) and reading the keyboard ID. This approach adds a fallback mechanism that retries the probe command or assumes a default ID for the keyboard (0xab83) if the keyboard does not respond or responds with an invalid ID. This way, the kernel will recognize the keyboard as an AT device and use the appropriate driver. This approach also uses the same kernel parameter, i8042.forceat2, to enable this modification. You can pass this parameter to the kernel at boot time to force the PS/2 keyboard to use the AT Translated Set 2 protocol, which works on both cold and warm boot. Here is what the code for this approach looks like:

--- a/drivers/input/serio/i8042.c
+++ b/drivers/input/serio/i8042.c
@@ -38,6 +38,7 @@
 #define I8042_KBD_IRQ		1
 #define I8042_AUX_IRQ		12
 
+#define I8042_CMD_SET_AT2	0xF0
 #define I8042_CMD_GETID		0xF2
 #define I8042_CMD_AUX_LOOP	0xD3
 #define I8042_CMD_AUX_SEND	0xD4
@@ -105,6 +106,7 @@ static bool i8042_bypass_aux_irq_test;
 static bool i8042_check_reset;
 static bool i8042_dritek;
 static bool i8042_dumbkbd;
+static bool i8042_forceat2;
 static bool i8042_noaux;
 static bool i8042_nokbd;
 static bool i8042_nomux;
@@ -122,6 +124,7 @@ module_param_named(bypass_aux_irq_test, i8042_bypass_aux_irq_test, bool, 0);
 module_param_named(check_reset, i8042_check_reset, bool, 0);
 module_param_named(dritek, i8042_dritek, bool, 0);
 module_param_named(dumbkbd, i8042_dumbkbd, bool, 0);
+module_param_named(forceat2, i8042_forceat2, bool, 0);
 module_param_named(noaux, i8042_noaux, bool, 0);
 module_param_named(nokbd, i8042_nokbd, bool, 0);
 module_param_named(nomux, i8042_nomux, bool, 0);
@@ -1070,6 +1073,16 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 	if (retval)
 		return retval;
 
+	if (i8042_forceat2) {
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		dbg("%02x -> i8042 (kbd get id)", I8042_CMD_SET_AT2);
+		i8042_write_data(I8042_CMD_SET_AT2);
+		i8042_write_data(0x02);
+	}
+
 	retval = i8042_wait_write();
 	if (retval)
 		return retval;
@@ -1081,6 +1094,15 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 	if (retval)
 		return retval;
 
+	if (i8042_forceat2 && !port->exists) {
+		dbg("keyboard did not respond to probe command, retrying");
+		retval = i8042_wait_write();
+		if (retval)
+			return retval;
+
+		i8042_write_data(I8042_CMD_GETID);
+	}
+
 	if (port->exists) {
 		retval = i8042_wait_read();
 		if (retval == 0) {
@@ -1094,6 +1116,12 @@ static int i8042_kbd_get_id(struct i8042_port *port)
 			}
 		}
 	}
+
+	if (i8042_forceat2 && !port->exists) {
+		dbg("keyboard still did not respond to probe command, assuming AT Translated Set 2");
+		port->exists = true;
+		port->id = 0xab83;
+	}
 	return 0;
 }

This code adds a conditional statement that checks if the i8042.forceat2 kernel parameter is enabled. If the condition is true, it sends the command 0xF0 to the keyboard port, followed by the argument 0x02, which sets the keyboard to use the AT Translated Set 2 protocol. This code is executed before sending the probe command (0xF2) to the keyboard. This code also adds another conditional statement that checks if the i8042.forceat2 kernel parameter is enabled and if the keyboard does not exist (port->exists is false). If the condition is true, it retries the probe command (0xF2) or assumes a default ID for the keyboard (0xab83) if the keyboard does not respond or responds with an invalid ID. This code is executed after sending the probe command (0xF2) to the keyboard. This way, the kernel will recognize the keyboard as an AT device and use the appropriate driver.