intel-precise-touch
intel-precise-touch copied to clipboard
linux-surface
IPTS on Surface Gen7
On the surface gen7 devices, IPTS has undergone several changes:
The binary firmware that was used for previous models does not exist anymore. Due to the changes in the GuC firmware it seems like Intel / Microsoft are now forwarding the data to userspace(?) using HID reports, and parsing it using a daemon (or another driver?). The HID descriptor is also not loaded from a binary anymore and needs to be dumped in Windows to analyze it.
The SET_MODE command will not accept any parameter except for IPTS_SENSOR_MODE_MULTITOUCH. Other parameters will result in the status IPTS_ME_STATUS_INVALID_PARAMS being returned.
Even though we can only initialize the device in "multitouch" mode, once the initialization was finished, the only that that is being sent are HID reports: the ones for singletouch, as well as a new report (with report ID 07). Other than that the device behaves exactly expected for multitouch mode (doorbell is incremented instead of signaling new data via. MEI responses).
This issue will be for tracking the progress on full IPTS support for gen7 surface devices.
We suspect that the new IPTS needs HID feature reports to enable proper multitouch mode. The old driver from intel has some hints towards sending HID reports using a special type of feedback, but some quick tests didn't appear to be working. It also has no proper documentation so it is mostly just guessing how the different parts behave.
Another thing that has no been tested is simply not sending the SET_MODE event and directly sending SET_MEM_WINDOW. Maybe there was a change so that multitouch mode is the default, and sending anything will change it to singletouch mode? (i.e redefining SET_MODE as sth. like SET_SINGLETOUCH_MODE)
Example for the new 07 reports:
ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: Buffer 2
07 b6 46 16 00 00 00 00 00 00 0b 00 00 00 00 ff 00 74 00 04 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: Buffer 3
07 4c 5f 16 00 00 00 00 00 00 0b 00 00 00 00 ff 00 74 00 04 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: Buffer 4
07 e2 77 16 00 00 00 00 00 00 0b 00 00 00 00 ff 00 74 00 04 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Patch for not sending SET_MODE (untested):
From af13ea8a753cbe3f4e366a8a927c5f1892360843 Mon Sep 17 00:00:00 2001
From: Dorian Stoll <[email protected]>
Date: Wed, 3 Jun 2020 15:41:15 +0200
Subject: [PATCH] no set-mode
---
receiver.c | 25 ++-----------------------
1 file changed, 2 insertions(+), 23 deletions(-)
diff --git a/receiver.c b/receiver.c
index ab28399..9884750 100644
--- a/receiver.c
+++ b/receiver.c
@@ -51,7 +51,8 @@ static void ipts_receiver_handle_get_device_info(struct ipts_context *ipts,
static void ipts_receiver_handle_clear_mem_window(struct ipts_context *ipts,
struct ipts_response *msg, int *cmd_status, int *ret)
{
- struct ipts_set_mode_cmd sensor_mode_cmd;
+ int i;
+ struct ipts_set_mem_window_cmd cmd;
if (msg->status != IPTS_ME_STATUS_TIMEOUT &&
msg->status != IPTS_ME_STATUS_SUCCESS) {
@@ -68,25 +69,6 @@ static void ipts_receiver_handle_clear_mem_window(struct ipts_context *ipts,
ipts->status = IPTS_HOST_STATUS_RESOURCE_READY;
- memset(&sensor_mode_cmd, 0, sizeof(struct ipts_set_mode_cmd));
- sensor_mode_cmd.sensor_mode = ipts->mode;
-
- *cmd_status = ipts_control_send(ipts, IPTS_CMD(SET_MODE),
- &sensor_mode_cmd, sizeof(struct ipts_set_mode_cmd));
-}
-
-static void ipts_receiver_handle_set_mode(struct ipts_context *ipts,
- struct ipts_response *msg, int *cmd_status)
-{
- int i;
- struct ipts_set_mem_window_cmd cmd;
-
- if (msg->status != IPTS_ME_STATUS_SUCCESS) {
- dev_err(ipts->dev, "0x%08x failed - status = %d\n",
- msg->code, msg->status);
- return;
- }
-
memset(&cmd, 0, sizeof(struct ipts_set_mem_window_cmd));
for (i = 0; i < 16; i++) {
@@ -206,9 +188,6 @@ static int ipts_receiver_handle_response(struct ipts_context *ipts,
ipts_receiver_handle_clear_mem_window(ipts, msg,
&cmd_status, &ret);
break;
- case IPTS_RSP(SET_MODE):
- ipts_receiver_handle_set_mode(ipts, msg, &cmd_status);
- break;
case IPTS_RSP(SET_MEM_WINDOW):
ipts_receiver_handle_set_mem_window(ipts, msg, &cmd_status);
break;
--
2.26.2
My earlier experiments with sending HID GET_FEATURE reports using host2me feedback:
https://gist.github.com/StollD/89f1c8d094c41e6ab5e3bbbe0dd0335b
Raw HID requests using host2me (aka. hid2me) feedback from the old Intel driver:
https://github.com/ipts-linux-org/ipts-linux-new/blob/master/drivers/misc/ipts/ipts-hid.c#L221
Unfortunately it does not work without SET_MODE on Surface Book 3:
[ 171.925322] ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: Probing IPTS
[ 171.925324] ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: IPTS using DMA_BIT_MASK(64)
[ 171.928265] ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: Device 045E:09B1 found
[ 171.930352] ipts 0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04: 0x80000003 failed: 1
@StollD I've set up qemu windows 10 and forwarded IPTS device to windows 10
I can see data flowing when I touch screen, vfio_region_read / vfio_region_write calls. Could you probably give me a hint what should I look. raw HID protocol in this writes?
Nice!
We want to figure out the initialization sequence that the new IPTS uses, so we need to look for that. It will be encapsulated by an unknown amount of wrapper data, because IPTS doesn't talk to the PCI device that you stubbed directly, it talks to the virtual MEI bus. There is a IPTS specific UUID, so we might be able to look for that.
Can you upload an example of the calls that are logged? Never messed with this myself, would be useful to know how that looks like.
Sure. This one is unprocessed, it should contain both initialization and several pen/finger touches
Alright, there are some interesting things.
https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3049-L3053
This is the command that sets the mode of the IPTS hardware. The parameters are all zeros, which, on the old IPTS, meant singletouch mode. If the first zero would be a one, it would be multitouch mode.
If windows initializes the hardware in singletouch mode and gets multitouch events, and we initialize it in multitouch mode and get singletouch, it might be that Intel swapped the meaning of these two values. However, the device otherwise seems to behave exactly like the old singletouch mode: It doesn't signal a new report by increasing the number in the doorbell buffer but uses the READY_FOR_DATA event.
For example:
- The driver signals that it is ready to receive data: https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3360
- The driver receives data and answers by sending feedback: https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3380-L3384
One issue with that theory though is that we already tested it: When the SL3 was just released, @archseer and I tried exactly that (swapping the meaning of single- and multitouch). The hardware returned an invalid-params error. Though it is possible that that was changed again in the SB3, or in a firmware revision. I also don't know what @archseer exactly changed back then, so it probably won't hurt to try it again.
https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L2874-L2878
IPTS sends a 0x9 command as part of the initialization sequence. According to the old driver from intel, 0x9 means SET_POLICIES. It's parameters doesn't seem to have anything to do with singletouch vs. multitouch, but there are some reserved fields, so maybe it was expanded to cover this.
- https://github.com/ipts-linux-org/ipts-linux/blob/master/drivers/misc/itouch/mei-itouch-messages.h#L81
- https://github.com/ipts-linux-org/ipts-linux/blob/master/drivers/misc/itouch/mei-itouch-messages.h#L234-L246
https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3001-L3007
There is a new 0xf command that is not documented in the old intel driver. Could be related, although I doubt it. It seems to carry a memory address as a parameter.
Feedback sending is...weird. It only seems to be sent for buffer 0xA and 0x2, and there are two instances of "special" feedback (with buffer ID 0x10).
- https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3284-L3288
- https://gist.github.com/StollD/e2fa122521bef3241782ef827285d998#file-trace-001-L3380-L3384
That special feedback could be used to send HID feature reports to the hardware, but there is no way for us to know, because these are stored in a seperate buffer and don't go over the PCI bus. But we also don't know if that special feedback is new, because we have no trace from a gen4-6 device to compare against.
@StollD I can get the same trace from gen6 (I believe) device (surface book 2) if it helps
Also if it will help in research I could record session with "pen only", "multitouch" and "touch"
@StollD I can get the same trace from gen6 (I believe) device (surface book 2) if it helps
That would be great actually! I was planning to research how to setup the passthrough myself, but if you already know how it works, it will probably be faster.
Also if it will help in research I could record session with "pen only", "multitouch" and "touch"
Not really. What I call multitouch covers all three of these. It is simply how we (and intel) call the mode that IPTS runs in. And windows doesn't allow changing it AFAIK.
Regarding my multitouch vs. singletouch theory: It should actually be enough if you loaded the master branch of this repo with the singletouch option set (which is also compiled into our builds already).
$ sudo modprobe -r ipts
$ sudo modprobe ipts singletouch debug
Then simply try to use the pen (or touch the screen), and watch dmesg.
You actually need to use the master version though, the UAPI version for iptsd won't work, because it doesn't support changing the mode.
@StollD unfortunately no reaction on pen / multitouch with singletouch option (i see logs on single touches so it was loaded correctly)
Out of curiosity could you post the dmesg? Maybe I can find some clue in there.
If simply changing the parameter for SET_MODE doesn't work, I think the next step would be to look at that SET_POLICY command a bit more. I can write a patch so that the driver will send exactly the command you captured during initialization.
Sure! dmesg.log
I've also tried to record another very short session trace on qemu, touching device only with pen, that 0x10 are still there, exactly 2 :(
trace.log.gz
I will be able to get logs from Surface Book 2 somewhere this week
Hmm. I'm actually very confused of size for trace.log.gz - it was a very short session of qemu (less than a minute). Why is it so big :/
I've also tried to record another very short session trace on qemu, touching device only with pen, that
0x10are still there, exactly 2 :(
Well, its part of the device initialization. That doesn't care about what tool you want to use on the device. ;) The actual device data is passed using memory buffers, and not through the PCI interface, so you won't see a difference if you change the tool (stylus / finger).
Hmm. I'm actually very confused of size for
trace.log.gz- it was a very short session of qemu (less than a minute). Why is it so big :/
The IPTS interface is just one function of the PCI device you are tracing. It could provide other interfaces that are used by windows that we don't really know about.
But I am confused about a thing too: Why is the linux driver logging singletouch events, if you added the singletouch option? Changing the mode should either produce an error (like when @archseer tried), or change the behaviour. IPTS shouldn't just ignore it.
Double checked again, master branch built, installed via dkms, specified in modprobe.d/ipts.conf
options ipts singletouch=1 debug=1
I see logs, I see reaction to single touches, no pen, no multitouch
Ok. Maybe the parameter has actually lost all its meaning.
Btw, I guess I know why the feedback that was sent looks weird to me. On linux we often had issues, especially on older devices, where sending feedback too often would cause the device to crash. Maybe on windows they only send feedback for every eight input (every time you read from buffer 2 or 10) to mitigate these issues and prevent bad firmware from crashing the device.
That would actually be a pretty smart thing to do, so we might want to replicate that behaviour under linux, for all devices.
I wrote a patch that sends the same SET_POLICIES command that is being sent in your trace. It seems to work fine on SB2. I would be curious what would happen if you apply this to the master branch, and load the module with the debug parameter set?
diff --git a/control.c b/control.c
index 9179eca..1bc00a0 100644
--- a/control.c
+++ b/control.c
@@ -59,6 +59,7 @@ int ipts_control_send_feedback(struct ipts_context *ipts,
int ipts_control_start(struct ipts_context *ipts)
{
+ u8 data[16];
ipts->status = IPTS_HOST_STATUS_INIT;
if (ipts_params.singletouch)
@@ -66,7 +67,12 @@ int ipts_control_start(struct ipts_context *ipts)
else
ipts->mode = IPTS_SENSOR_MODE_MULTITOUCH;
- return ipts_control_send(ipts, IPTS_CMD(NOTIFY_DEV_READY), NULL, 0);
+ memset(data, 0, 16);
+ data[0] = 0x12;
+ data[2] = 0x64;
+ data[4] = 0x1E;
+
+ return ipts_control_send(ipts, IPTS_CMD(SET_POLICIES), data, 16);
}
void ipts_control_stop(struct ipts_context *ipts)
diff --git a/protocol/events.h b/protocol/events.h
index f8b771f..b50d399 100644
--- a/protocol/events.h
+++ b/protocol/events.h
@@ -24,6 +24,7 @@ enum ipts_evt_code {
IPTS_EVT_FEEDBACK,
IPTS_EVT_CLEAR_MEM_WINDOW,
IPTS_EVT_NOTIFY_DEV_READY,
+ IPTS_EVT_SET_POLICIES,
};
#endif /* _IPTS_PROTOCOL_EVENTS_H_ */
diff --git a/receiver.c b/receiver.c
index ab28399..c0f60c2 100644
--- a/receiver.c
+++ b/receiver.c
@@ -10,6 +10,20 @@
#include "protocol/responses.h"
#include "resources.h"
+static void ipts_receiver_handle_set_policies(struct ipts_context *ipts,
+ struct ipts_response *msg, int *cmd_status)
+{
+ if (msg->status != IPTS_ME_STATUS_SENSOR_FAIL_NONFATAL &&
+ msg->status != IPTS_ME_STATUS_SUCCESS) {
+ dev_err(ipts->dev, "0x%08x failed - status = %d\n",
+ msg->code, msg->status);
+ return;
+ }
+
+ *cmd_status = ipts_control_send(ipts,
+ IPTS_CMD(NOTIFY_DEV_READY), NULL, 0);
+}
+
static void ipts_receiver_handle_notify_dev_ready(struct ipts_context *ipts,
struct ipts_response *msg, int *cmd_status)
{
@@ -196,6 +210,9 @@ static int ipts_receiver_handle_response(struct ipts_context *ipts,
int ret = 0;
switch (msg->code) {
+ case IPTS_RSP(SET_POLICIES):
+ ipts_receiver_handle_set_policies(ipts, msg, &cmd_status);
+ break;
case IPTS_RSP(NOTIFY_DEV_READY):
ipts_receiver_handle_notify_dev_ready(ipts, msg, &cmd_status);
break;
No effect unfortunately. Still modprobes, still reports single touches only :cry: (with or without singletouch option)
Damm. I had really hoped that would work. The other new command (the 0xf one) seems to contain a memory address, so sending that ourselves might not be that simple.
It's either that, or changing the modes really happens through special feedback, which will be pretty hard for us to replicate.
Sad. I'm trying to get myself familiar with dtrace for windows introduced by Microsoft not so long ago - it works extremely low level so theoretically it should be possible to trace special feedback function calls in windows
Additionally, I've put iaPreciseTouch.sys in my IDAPro64 and confused - I was expecting to see number of HID functons, so theoretically it will be possible to debug them with kernel debugger, ~~but the only imported id function is HidNotifyPresence :/~~
I see IOCTL_HID_WRITE_REPORT invocation, so it definitely uses that one
Sorry for "blogging". It seems that initial assumption about additional DLL for processing is correct. Surface is installing TouchPenProcessor.dll which is after quick look in disassembly imports non-typical for driver math functions (like log, pow, etc.) and it is named like "HEAT-compliant touchscreen" and has functions like CreateHeatProcessor
Unfortunately this DLL also includes just HidP_GetCaps and HidP_GetValueCaps calls
Some new dumps from surface book 2
The first one is pretty funny. I've just copied qemu image with my surface book 3 driver to surface book 2. It initialized, but reported only single touches. Also I've noticed that same updates arrive when we're in (for example) boot sequence, so it seems like it's just a default mode for touch screen trace-sb2-with-sb3-driver.gz
Second one is with sb2 driver trace-clean.gz
Thank you!
Your trace-clean log confuses me a bit. Were you able to use the pen with that? Because the commands in the log are the ones that are used for singletouch.
I have played with it a bit myself, and I don't think it will be possible to enable multitouch mode on gen4-6 within the VM, at least not with a custom build of QEMU.
Multitouch mode on the old IPTS depends on binary firmwares that are run on the GPU. The firmware to use is selected by querying the name of the \_SB.TSML ACPI device, which does not exist on the VM. We would need to hack it in there by modifying qemu.
But even then it will probably not work, because we don't pass through the GPU, so the driver won't be able to process touch data there.
Both, the TSML ACPI device, and the touch processing on the GPU are not used on gen7 though. So it should be less of an issue there.
Just to be sure, @xanf when you tried it on gen7, was the hardware actually in multitouch mode? If it is in multitouch mode, either the pen works, or nothing works (because no touch processing driver is installed). If it detects single touches, and nothing else, it is in singletouch mode, which would match the results from our earlier experiments on linux.
Also, I would be curious about how your VM was set up. I just made a generic windows install, and tried to install the surface driver package. But that refused to run, because it didn't detect an actual SB2. I resorted to unpacking the installer and installing the drivers by hand, but there are so many, and we have no idea what is really needed. Also, at least on SB2, the installer seems to do some more setup in the registry, so we loose that too.
I am wondering if we might get better results if we create a disk image of an actual windows install, where all these drivers are installed properly, and use that for the VM. I've seen that microsoft has recovery images for the surfaces, maybe that can also be of use.
On surface book 3 system was reacting with reads / writes in log to pen, I've checked this
I've also resorted to manual installation of unpacked driver, for surface book 3 it seems two devices are sufficient - intel precise touch driver (and installing this alone enables touchscreen to "receive" pen events and something ( i don't remember exact name, does not have sb3 right now with me) called surface touch processor - it was exactly the device which had non-typical log, sin, cos functions in disassembly.
I've figured required driver by disabling them one by one in windows and checking when touch stops working
I think on the SB3, access to the GPU shouldn't be required as it looks like touch processing happens in the TouchPenProcessor.dll (the one with the log/sin/cos functions). On the SB2, the iGPU is required but I think (with some amount of work) it should be possible to pass it through via IOMMU, at least on the units with a dGPU. Probably requires you to run either no desktop at all or the desktop via the dGPU on the host (not sure if it's even possible without a dGPU, this seems to indicate it is).
Regarding ACPI, its possible to add/load additional ACPI tables in qemu (should be -acpitable parameter), so you don't have to modify qemu directly. However, you might not be able to directly copy the ACPI table and have to just write the relevant parts of it yourself instead (in case of dependencies etc.).
On surface book 3 system was reacting with reads / writes in log to pen, I've checked this
@xanf Sorry if I sound like a broken record. But did the pen work (inputs getting registered inside the VM)? Alternatively, did nothing work, not even single touches?
Basically, I want to figure out if the driver in the VM runs actually runs in multitouch mode, or in singletouch mode. Because if it is in singletouch mode, it wouldn't be suprising that no command that we find in the trace logs helps with getting into multitouch mode.
Sadly just looking at the trace doesn't really help in figuring this out. The IPTS commands are just a very small part of the trace log. And even then, they could be anything. You need to check how the driver behaves. And using the pen is a good test for that, because these events will only arrive in multitouch mode. So if the pen works, and the VM picks up the inputs, its in multitouch mode.
Same for "no input at all": This would mean that the driver sets the device to multitouch mode, but then some other part of the data processing stack for multitouch is missing, so that the inputs cannot be processed. Since we only care about the PCI trace, this would be good enough too.
But if only single touches are registered, the device is in singletouch mode, and the trace logs won't help us with getting the device into multitouch mode. In that case we would have to install additional drivers, until something changes.