intel-precise-touch
intel-precise-touch copied to clipboard
linux-surface
Support for Touch Host Controller (Tigerlake+)
Hi,
I know this repo is for surface devices, but the new Intel NUC M15 Laptop seems to use the same touchscreen controler. Windows recognizes it as Intel Precision Touch & Stylus or similar (currently booted on Linux but can look up the exact name if it becomes relevant). Linux sudo lspci -vvnn:
00:10.6 Digitizer Pen [0901]: Intel Corporation Device [8086:a0d0] (rev 20)
DeviceName: Onboard - Other
Subsystem: Intel Corporation Device [8086:2097]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
IOMMU group: 10
Region 0: Memory at 603e1d0000 (64-bit, non-prefetchable) [disabled] [size=32K]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
I'm on linux-surface kernel 5.10 which correctly recognizes the MEI:
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-LP Management Engine Interface [8086:a0e0] (rev 20)
DeviceName: Onboard - Other
Subsystem: Intel Corporation Device [8086:2097]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 156
IOMMU group: 15
Region 0: Memory at 603e1ed000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00638 Data: 0000
Capabilities: [a4] Vendor Specific Information: Len=14 <?>
Kernel driver in use: mei_me
Kernel modules: mei_me
However regardless of using the dkms or the builtin driver the ipstd service crashes on start:
● iptsd.service - Intel Precise Touch & Stylus Daemon
Loaded: loaded (/usr/lib/systemd/system/iptsd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2021-02-18 12:49:47 CET; 5h 30min ago
Docs: https://github.com/linux-surface/iptsd
Process: 1361 ExecStart=/usr/bin/iptsd (code=exited, status=237/KEYRING)
Main PID: 1361 (code=exited, status=237/KEYRING)
Feb 18 12:49:42 test-pc systemd[1]: Started Intel Precise Touch & Stylus Daemon.
Feb 18 12:49:47 test-pc iptsd[1361]: ERROR: ../src/control.c:52: Failed to send feedback: No such device
Feb 18 12:49:47 test-pc iptsd[1361]: ERROR: ../src/control.c:75: Failed to flush buffers: No such device
Feb 18 12:49:47 test-pc iptsd[1361]: ERROR: ../src/main.c:83: Failed to start IPTS: No such device
Feb 18 12:49:47 test-pc systemd[1]: iptsd.service: Main process exited, code=exited, status=237/KEYRING
Feb 18 12:49:47 test-pc systemd[1]: iptsd.service: Failed with result 'exit-code'.
Could it be that the Surface devices use the same controller, but connect it differently than the NUC M15? Some hints that lead me to this theory:
- The surface devices seem to not have the Digitizer Pen PCI device (I have no Surface device, so my source is linux-hardware.org)
- I extracted the driver package for the NUC M15 as well as one of the Surface devices supported by this repo. I grep'd for strings found in the NUC Touchscreen driver .inf file but could not find any hints for this driver in the Surface driver package. My Theory: While on the Surface devices the Touchscreen controler is managed by the ME (which this driver is for) und the NUC M15 the controler is directly accessed over the pci bus, which would require a completly differend driver.
I hope I'm wrong ^^.
My Theory: While on the Surface devices the Touchscreen controler is managed by the ME (which this driver is for) und the NUC M15 the controler is directly accessed over the pci bus, which would require a completly differend driver.
This is pretty much correct. With Tiger Lake, Intel deprecated the old MEI based IPTS and instead moved its functions to a standalone PCI device, the Touch Host Controller (THC). Some Lenovo device is out there using it too.
Since the THC is not MEI based, I doubt this driver can be reused at all. It might be possible to reuse the iptsd daemon, if the data that the hardware outputs is the same or similar. But there is no way to be sure without having a working kernel driver.
What I find interesting is that the IPTS driver probes on your device and creates its userspace interface. That either indicates that some life is left in the old interface, or that Intel adds the interface to all of their hardware (which would be annoying for upstreaming this driver). Could you post your full dmesg?
I've had a quick look over the driver package for that device. There's a iTouch THC Base Driver, but that doesn't seem to have any firmware package like the old IPTS driver. There are some interesting strings in there (GetOutputHidXferPacketFromRequest, FlashlessFwDataFile, FlashlessFwStage1File, ...), not sure what to make from that yet.
There's a touchpad driver with firmware (but I think that's really for the touchpad and not the touchscreen, seems to be a PixArt PCT1836 device and it does look like it's a touchpad controller similar to this one).
There's also a HID Event Filter driver. There are references to some ACPI devices (INT33D5, INTC1051, ...) in a config file associated with that, but I couldn't find anything on them. The strings in that driver do indicate that it's for some function keys though (wireless radio button, home button, volume buttons, ... are all referenced)
So I'm wondering how that works exactly. There doesn't seem to be any obvious firmware around (unless that's baked into the hardware and just needs to be updated when it changes or baked into some driver). And unless the HID Event Filter thing is doing touch processing (which I kinda doubt due to the ACPI references), there doesn't seem to be any equivalent of a TouchPenProcessor.dll around either. So maybe that's distributed separately or already integrated into Windows by now?
Also having this as a PCI device without any way of controlling touch processing (be it GuC/compute kernels or user-space processing) doesn't make much sense to me. If it'd be just another touch controller that needs some lockded-down firmware for processing, why not use I2C, SPI, USB or whatever.
Well, the IPTS protocol can also transport HID reports instead of raw sensor data. I wouldn't rule it out that an OEM could do the processing in the touch controller firmware and only use IPTS for passing the data around.
Yeah, but Intel themselves doing that? I'd think the NUC devices would be a showcase of sorts for them to demonstrate how things work. Maybe chipset documentation could give us a hint.
There's not much more than "Touch Host Controller provides the standard SPI interface to connect to external touch ICs.". So maybe it really doesn't do processing.
Oh hey, apparently there's a patent application for it: US20190102032A1.pdf. Doesn't really clear things up for me though.
What I find interesting is that the IPTS driver probes on your device and creates its userspace interface.
I rechecked and realized that the /dev/ipts/X devices are only there when I manually modprobe the ipts driver. The iptsd was started after that. When I start it before, I just get a "No such file or directory error".
dmesg after boot: unloaded.txt
dmesg after modprobe ipts loaded.txt
I use the linux-surface kernel build from here https://github.com/linux-surface/linux-surface/wiki/Installation-and-Setup#arch :
Linux test-pc 5.10.16-arch1-2-surface #1 SMP PREEMPT Mon, 15 Feb 2021 20:59:59 +0000 x86_64 GNU/Linux
and have additionally installed the dkms ipts driver via the AUR.
I rechecked and realized that the /dev/ipts/X devices are only there when I manually modprobe the ipts driver. The iptsd was started after that. When I start it before, I just get a "No such file or directory error".
Ah, I didnt think about manually loading the driver. That confirms that the MEI interface is gone then. It doesnt really help you, but thanks nonetheless.
Thanks from my side for the quick response ^^
Now at least I know that I don't need to test further with this driver.
The new Surface Laptop Studio quite likely also uses this interface. See archive in https://github.com/linux-surface/acpidumps/issues/20#issuecomment-939340816 for lspci output.
I am getting a SP7+ for BF and I was looking into installing EndeavourOS on it, then I saw this issue.
It seems the THC is going to end up in all next Surface line products. Almost all the new lineup (except the SLGO and GO3) is having it. If intel is pushing this in future hardware, does this mean that we are going to see more and more products missing touchscreen on linux ? Is Intel touchscreen related hardware widespread amongst the industry or is it confined to specific devices such as the Surface line ?
The NUC laptop lines is often used to be integrated by OEM companies like Schenker (XMG, tuxedo, etc..) . Some of them are quite open-minded about linux support so I am wondering if they are talking with Intel about THC support in Linux (especially releasing a driver or technical documentation to build it)
For instance the XMG Fusion 15 (https://download.schenker-tech.de/package/xmg-fusion-15-xfu15l19/) is based on the previous NUC laptop version and they ended up officially supporting it on Linux (despite the fact Intel never released a driver pack for it https://www.intel.com/content/www/us/en/products/sku/203496/intel-nuc-9-extreme-laptop-kit-lapqc71c/downloads.html) The Schenker Vision 15 is based on the NUC M15 (https://download.schenker-tech.de/package/schenker-vision-15-svs15e21/) however they do not support linux. Again intel did not release linux driver pack for it (https://www.intel.com/content/www/us/en/products/sku/205490/intel-nuc-m15-laptop-kit-lapbc710/downloads.html)
It seems the THC is going to end up in all next Surface line products.
The Intel ones, yeah.
If intel is pushing this in future hardware, does this mean that we are going to see more and more products missing touchscreen on linux? Is Intel touchscreen related hardware widespread amongst the industry or is it confined to specific devices such as the Surface line?
At the moment I don't think there are many devices apart from the Surface lineup (and maybe the Intel demo ones you mentioned) that use it. As for the future, I guess only Intel and/or device manufacturers can really answer that.
In contrast to the old MEI interface that this seems to replace, the THC doesn't seem to need any processing kernels and AFAICT could be used with any ordinary SPI touchscreen controller (which seems to be the standard). As I understand it, it's just a SPI interface optimized for lower power consumption and touch usage. So the cost of adoption is a lot lower and adoption may even come with benefits, which may make this interesting to device vendors. The only reason to not choose this over other interfaces would probably be the Linux support, so if I understand this correctly there's a good chance that other manufacturers will adopt this.
Some (mostly Lenovo and Dell) might not due to missing Linux support, which might incentivize Intel to actually release a driver for that in some way or another. I guess if Intel is serious about pushing this technology they will eventually provide a driver (to get manufacturers to use that interface), but combine touch and Linux and you have a very small user-base that is affected, so for them it's probably not a high priority.
Regarding driver packages and Linux: That's really not how drivers work in Linux. Everything is supposed to be in the kernel, without the need for external modules (that have an unstable API which may break at any time). So if they are serious about Linux support they will make an effort of upstreaming a driver. However, keep in mind that Intel's Linux support is generally datacenter first, consumer second (or not at all).
You''re right about the userbase it can be quite thin. However as other intel hardware such as Wifi/BT cards, iGPU are supported (especially because they are widespread), I keep hope...
You are right about the driver package, I was mentioning the windows one because I thought it could help as a reference for future implementation/reverse engineering. The Surface BIOS handles the touchscreen input (I guess a quite basic one) so I think that driving this hardware to got at least basic touchscreen support can be attainable without huge processing in the driver/daemon.
I am not a specialist about those touchscreen devices but from what I read about your analysis, do you mean the THC outputs commands that are more low-level than the old MEI ones ?
I am not a specialist about those touchscreen devices but from what I read about your analysis, do you mean the THC outputs commands that are more low-level than the old MEI ones?
The MEI interface was designed around a touchscreen that provides raw data which is then processed via GuC on the iGPU. So that's a lot more complex and vendors need to write/provide a firmware package that does that processing. The THC just seems to be another communication channel (at least according to their marketing slides), so vendors should be able to just plug in any existing OEM touch controller with an SPI interface.
I posted a question on intel forums and got a response: https://community.intel.com/t5/Intel-NUCs/Touch-Host-Controller-Linux-support/m-p/1339579
TLDR: not gonna happen any time soon....
Interesting thing I noticed. On a SP7+ if you use refind and set "enable_touch" touch screen works on refind. I guess the hardware is working in a sort of legacy mode when in EFI.
@aacebedo In the "old" drivers there are essentially two modes of operation: Single-touch/simple and multi-touch. Single-touch provides just single-touch information without any pen support. Multi-touch provides a touch heatmap/proprietary data that needs to be processed either in kernel or in user-space to make sense of things and extract touch/pen information. I'm fairly certain that the new driver is just a communication channel to the touch controller, so I expect it to work in the same way.
The Surface UEFI provides a driver for the single-touch mode. That's essentially active during the pre-boot environment and can be used by the UEFI menu and any bootloader, like refind. It's basically the reason why the Surface devices support touch in UEFI. Unfortunately that driver gets unloaded when we switch to the OS/kernel mode, i.e transition out of the UEFI pre-boot environment.
Things don't work that way. Think of UEFI as another (although limited) operating system next to Windows and Linux. Just like Windows drivers don't work with Linux, the UEFI drivers don't work with Linux.
You could hack out the transition from pre-boot to OS mode in the kernel, but that would only ensure that the UEFI drivers stay loaded. You still can't use them (the only thing that helps you with is that now some of the hardware may be set up and configured for use, e.g. clocks and voltages). But then you have potentially two sets of drivers accessing the same hardware which in the best case leads to just instabilities.
I've done a bit of reverse engineering, and currently have a semi working polling-based single-touch THC driver (tested on a Surface Pro 7+). I'm working on converting it to using interrupts, getting the multi touch data out, and figuring out some other details. Will probably put something online somewhere during the next few weeks, depending on how much free time I have.
If it's spitting out HID data (which I assume) then getting multi-touch and pen data will probably only require you to send a HID feature report to the device. That's at least how the other (non GuC-processing) devices worked. So in essence, if it works with single-touch, it should also work multi-touch, no driver changes required.
Interestingly the THC actually has two identical DMA receive interfaces. The second one sends HID data, so I thought maybe the first would allow access to the raw data somehow, especially since there's a function that seems to hook it up to the GuC. I spent some time poking at it, but I haven't been able to get that first interface to give me any data. And the GuC function doesn't actually seem to get invoked by the driver or any other driver that I've looked at.
After I gave up on that, I saw you guys figured out the HID feature stuff in another issue, and yup, that actually works (but it still sends the multi touch data via the second DMA interface, leaving the first one unused).
What I still need to do is actually get the data into userspace somehow. I think your driver uses a chardev, right? I was thinking it may be possible to make a zero copy interface via mmap-able sysfs files.
AFAICR it has (according to the marketing slides I've seen) support for two separate touchscreens (this doc seems to support that). So makes sense that it has two DMA interfaces.
Yeah, we're currently using a custom chardev, but in the long term I think it's probably better if we go via a hid(raw) device. Data is already sent via HID, so why not use something more or less standard? That would also make the single-level stuff work without IPTSd/user-space processing, so you'd get an automatic fall-back mode if you don't have IPTSd active/installed. Mmap could probably get latency down slightly further, but I'm not sure if it's really worth the additional hassle.
Nice work!
I think an issue with the mmap approach was that the MEI driver is removed every time you go into suspend, but the chardev has to stay active, because iptsd is sometimes faster than the kernel and tries to read from it before it is back online. Copying the data seemed more robust and it works just fine.
Regarding the two interfaces, maybe one is strictly for commands and the other one is strictly for data? Kinda like the old IPTS has feedback and data buffers.
So here's the driver: https://github.com/quo/ithc-linux
And a slightly modified iptsd to go with it: https://github.com/quo/iptsd
You're probably right that a zero-copy interface isn't worth it. Mapping the DMA ringbuffer into userspace isn't that difficult, but I can't really see a clean and race-free way to notify userspace of new data.
So for now I just used a chardev which spits out the DMA RX data with a little header. When the device is opened/closed, the driver automatically enables/disables multitouch mode, so single touch automatically works when iptsd isn't running.
I added an optional HID mode. This sends the data through the HID subsystem. It works for single touch. It creates a hidraw device that iptsd could theoretically use for multitouch, but I haven't tested that. On the one hand HID seems like the right way to do this, since the device sends HID reports. But only the single-touch data is really HID... The multitouch data doesn't really benefit from being marshalled through the HID subsystem, it just seems a little pointless (and inefficient).
As for the two DMA RX interfaces, I like the theory that it could just be one for each SPI touchscreen chip. There's only a single SPI command interface and a single DMA TX interface though (but I guess one of the unknown bits could be a chip select). And the Intel driver seems to set the two interfaces up slightly differently. So I don't know.
Awesome! Thanks for sharing!
But only the single-touch data is really HID... The multitouch data doesn't really benefit from being marshalled through the HID subsystem, it just seems a little pointless (and inefficient).
Isn't the multitouch data already HID wrapped? The benefit of going full HID is that it'd be a simpler driver. Otherwise you'd have to parse HID records in the driver (or forward them via a custom interface, at which point you're more or less re-implementing hidraw), which I don't think is really necessary (we already have to process that data in user-space). That also has the benefit that the driver is more device independent. I think you can't expect non-MS devices to use the same non-HID data (on the contrary, if this driver is adapted, I think quite a lot of devices will likely use "standard" HID data). So adding custom handling code (including stuff like enabling multi-touch inside the driver) for that is more or less driver bloat for other devices. (Again, only if it's already HID-wrapped.)
Sure, there is some overhead of running this through the HID subsystem. But I think there are more benefits to going the HID route than trying something custom. HID is already well tested and accepted and the kernel code is (hopefully) somewhat optimized.
Isn't the multitouch data already HID wrapped?
Not really. I mean, sure, they've added a report ID byte at the front, but that's it. The report descriptor just says that report ID 11 is a 3500 byte blob of vendor specific data. Not very helpful. If this were really HID, the report descriptor would describe the structure of the data in the report, so that you could at least try to handle it generically.
I'm not necessarily opposed to using the HID subsystem, that's why I added the option. But Intel/MS's usage of HID just feels like a hack.
Yeah, I mean that's HID. Reports really are just a report ID + report specific data (and there's a ton of custom reports out there, not necessarily input but mostly custom feature stuff). And yeah, you can see the input reports as a hack. But it's still valid HID (and it even has a spec for the "wrapper": https://usb.org/sites/default/files/hut1_3_0.pdf 16.9, although that seems to be more or less misused on MS Surface devices as the heatmap data should probably have usage 0x6C and not 0x61).
I mean my main point is: The device (to me) seems like a HID communication channel. So the driver should (more or less) act like that (i.e. be a HID transport driver). Anything else (like handling MS custom heatmap data) belongs (from a design perspective) into some different driver (HID device driver) or user-space. That will likely also make it easier to upstream.
Also I should mention: I don't have any issues including this as-is for now. If we go with the HID route, the IPTS driver also needs changes (which are more substantial as on the older devices it's actually not HID data).