Custom HID device parsing (macOS HID interrupt buffer access)

[thatandyward]

I am trying to implement real-time access to a USB vibration sensor. The device continually writes raw acceleration values to the HID interrupt buffer.

Reading the interrupt buffer via the hid_read() function on macOS always returns 0 i.e. no packets available to be read; however, when I run the same script on Linux the hid_read() function returns data from the buffer just fine.

I'm wondering if there is something specific to macOS IOHIDManager which is preventing access to the interrupt buffer?

I don't think the HID device is being captured by the OS (like a keyboard) —I can successfully access various feature reports (getting/setting device time, reading parameter states etc). Although I'm not familiar with the details of how/when the OS captures HID devices so might be wrong. Can the OS capture some functions of a HID device, but not others?

I have used Wireshark to sniff the device traffic and can see the expected interrupt buffer data being streamed from the device. (Wireshark capture output attached below)

Looking at the device descriptor details (via lsusb, output attached below) I can see the device has 2 interfaces —Interface0 is a mass storage device (it has an internal SD card which mounts as a drive) and Interface1 is an HID device with a single endpoint (EP 3 IN) and an Interrupt transfer type. Which all seems to be as expected.

Playing with the timeout duration doesn't seem to have any effect. The interval for Interface1 is 4 frames (4ms).

I am using hidapi via a julia wrapper; however I also tried hidapitester to verify there was nothing funky happening in the wrapper/my script. I get the same results. I can open the device, read/send feature reports

% ./hidapitester --vidpid 03eb/6135 --open -l 7 --read-feature 1
Opening device, vid/pid: 0x03EB/0x6135
Reading 7-byte feature report, report_id 1...read 7 bytes:
 28 37 13 02 26 80 21
Closing device

however the --read-input-forever function just returns 0.

% ./hidapitester --vidpid 03eb/6135 --open -l 25 --read-input-forever 17
Opening device, vid/pid: 0x03EB/0x6135
Reading 25-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 25-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 25-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 25-byte input report 0, 250 msec timeout...read 0 bytes:

I have tried specifying the report number and leaving it blank, which doesn't have any effect.

Could the OS-level 'reading' of the interrupt buffer stream be blocking? Interestingly I am unable to get the hid_set_nonblocking() function to work —it does not return any exit code.

Appreciate any help/advice/pointers in what might be happening here and how I can get this working on macOS.

Thanks

~a — — — — wireshark capture.txt lsusb output.txt ioreg output.txt (device at lines 12 & 36)

[todbot]

What is the HID Report Descriptor for your device? That is usually more important than lsusb or ioreg output. On Linux you can use usbhid-dump for USB HID devices and on MacOS you can use the "USB Prober" app. (you can get it from developer.apple.com but I put a copy of USB Prober here too)

The HID Report Descriptor will tell you whether or not you should be using hid_read() or hid_get_feature_report() to receive data from the device in question. If your device isn't sending proper HID INPUT or FEATURE reports that abide by the HID Report Descriptor the OS may drop those reports. Or if the HID Report Descriptor describes that the device is using one of the OS-reserved HID Usages (e.g. keyboard, mouse, gamepad, etc), then the OS will "own" that device and not forward reports to hidapi.

[thatandyward]

Thank you for your reply, so helpful! And thank you for the USB Prober app, I have been searching for a way to access the report descriptors —this is super useful!

Looking at the report descriptors (USB Bus Prob output attached), both the application collection and all reports are listed as Usage 1 (0x1), which appears, from quickly skimming the HID usage tables, to be GENERIC DESKTOP PAGE. That seems to indicate it is not an OS-reserved HID usage —am I interpreting that correctly?

— —

The report descriptors seem to align with the (limited) documentation I have from the device manufacturer.

The feature ReportIDs listed correlate with the feature reports I am able to read via hid_get_feature_report()

There are also some ReportIDs listed which I believe are for different devices from the vendor, and not applicable to the device I have (temperature, humidity, pressure etc).

ReportIDs (6) and (17) are the two input reports which get written to the interrupt buffer when I'm able to read the buffer on linux. (6) is a PPS tick, which writes date/time once a second (17) is the raw accel data that gets written continuously

These are both listed as Input reports, although am not sure how to tell if they abide by the HID report descriptor. I have had a quick read of the Device Class Definition for HID 1.11 section 6.2.2 on report descriptors and nothing jumps out as being obviously incorrect, but I have not looked in detail. Is this the correct section?

I have tried accessing these reports via both the hid_read() function and the hid_get_input_report() function — the latter reports that it successfully wrote the correct number of bytes; however the passed byte array is filled with 0s when returned, with the exception of the reportID in the first address. When I pass a byte array filled with 1s they are overwritten with 0s when returned.

The usage type would seem to match with Dynamic Value (DV) as it's a constantly changing set of values time-offset, x_accel_val, y_accel_val, z_accel_val

Type	Flags	Description
Selector (Sel)	Array	Contained in a Named Array (NAry).
Static Value (SV)	Constant, Variable, Absolute	A read-only multiple-bit value.
Static Flag (SF)	Constant, Variable, Absolute	A read-only single-bit value.
Dynamic Value (DV)	Data, Variable, Absolute	A read/write multiple-bit value.
Dynamic Flag (DF)	Data, Variable, Absolute	A read/write single-bit value.

The report descriptors for both (6) and (17) seem to include Data, Variable, Absolute fields from what I can tell.

Is there a good way to determine if the HID INPUT report conforms to spec? Are there particular issues that typically cause the OS to drop the reports?

Thanks

~a — — — — USB Bus Probe.txt Report Descriptors lines 71-154

[todbot]

In the case of usagePage vs usage, usagePage is the "container" (e.g. within a usagePage are several usages), and your device in question is usagePage = 0xFF00, a "vendor-defined" usagePage, so the OS will ignore it. And since this is a vendor-defined usagePage, the primary usage = 0x01 it defines doesn't mean anything standard, and thus appears to not conform to any known spec. As a comparison, usagePage = 0x01 is "Generic Desktop" within that is usage = 0x02 (mouse). You can read about HID Usage Tables in this PDF.

From what I can see:

• ReportIds 1, 2, 3, 5, 7, 22, 19 appear to be FEATURE reports and so you should use get_feature_report() or send_feature_report(). FEATURE reports go either way, and there's no way to know which way without knowing what the device intends.

• ReportIds 17, 20, 8, 10, 11, 6, 12, 15, 9, 26, 21, are all INPUT reports so should be used with hid_read()

When sending receiving any kind of report:

• Make sure to request the appropriate number of bytes for each reportId (REPORT_COUNT + 1), e.g. reportId 21 has a REPORT_COUNT of 37, so ask for 37 + 1 = 38 bytes (+1 is for the reportId)
• This is a weird device and you may get back all zeros on an INPUT report if the device hasn't been configured (presumably via a SET FEATURE report) to fill out that report.

Does this device come with an SDK you can crib from? Often you can dig into the source of SDKs to find what's going on. Since you're doing Wireshark, if Wireshark has HID parsing, I would set it to that and only look at what FEATURE reports are being sent to configure the device and then what INPUT reports are being read.

[thatandyward]

Thanks @todbot, really helpful stuff.

So the "vendor-defined" usagePage rules out the possibility of the OS 'owning' the HID device, correct? i.e. not the reason the hid_read() function is unable to read the interrupt buffer on macOS for this device.

— — 

The vendor has a Linux command line tool for their range of devices which I believe is used to configure, test and upgrade firmware etc. The tool also has the ability to access raw data from the device.

They sent me the C source code and, whilst I'm not very familiar with C, I have been able to decipher enough that I have a pretty good understanding of the device reports and how to interpret the raw data returned from the device. So no issues there.

On Linux everything works as expected, the hid_read() function returns the HID INPUT reports that the device is generating (ReportIDs 6 and 17); However on macOS the hid_read() function always returns 0 i.e. no packets available to be read, which I think is incorrect as I know from Wireshark the device is generating reports.

Looking at the Wireshark capture on macOS, I can see both get_feature_report() and send_feature_report() submit/complete requests when my script sends them. These are URB_CONTROL in with an EP0 address, which all makes sense.

The device is generating a constant stream of HID INPUT reports (ReportIDs 6 and 17) with no input from my script. These show in the Wireshark capture continually when the device is plugged in. They are URB_INTERRUPT in submit/complete requests with an EP3 address occurring every +/-10ms for report 17 and every second report 6 (which is the expected timing). The raw data packets are valid and can be interpreted correctly.

When my script sends an hid_read() request it does not appear in the Wireshark capture at all —There are no submit requests without a corresponding complete request. And all the complete requests contain valid and complete data.

Interestingly when I look at the Wireshark capture on Linux it does not have the same constant stream of URB_INTERRUPT in submit/complete requests. When my script sends the hid_read() function it appears in the Wireshark capture as expected.

So it seems to me that on macOS the device is sending the contents of the interrupt report buffer to the host continuously, vs. Linux where the report needs to be proactively requested before being sent.

Could it be possible that because the device is continuously sending the interrupt buffer contents to the host that a) the buffer is always empty when polled? Or b) is always blocked because it's constantly 'being read'?

If a device is being blocked by another process would hid_read() return 0 or -1?

Thanks again for your help

~a p.s. thanks also for hidapitester, it's been really useful.

[todbot]

Some quick thoughts:

• Are you using the exact same code on Linux & Mac?
• Are you adding an extra byte to the length to account for the reportId byte (-l option in hidapitester)? I notice that in your above example you have -l 25 --read-input-forever 17 but since the device is using reportIds and report 17 is 25 bytes big, you should instead do -l 26. I think Mac is strict about the length but Linux is not.

[thatandyward]

yes, exact same code on Linux and macOS.

I am adding the extra byte —although I have tried playing around with that, which is why it was left set at 25 in my example!

macOS seems pretty forgiving, if the passed length is shorter it still returns, just truncated. If the passed length is too large it pads with trailing 0s.

Anyway, for completeness here is the hidapitester output for reportID 17 that includes the additional byte.

% ./hidapitester --vidpid 03eb/6135 --open -l 26 --read-input-forever 17
Opening device, vid/pid: 0x03EB/0x6135
Reading 26-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 26-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 26-byte input report 0, 250 msec timeout...read 0 bytes:
Reading 26-byte input report 0, 250 msec timeout...read 0 bytes:

[mcuee]

Not sure if this is an HIDAPI issue or macOS issue.

@thatandyward The following may not be the reason but no harm trying. https://nachtimwald.com/2020/11/08/macos-iohidmanager-permission-issue/

[Youw]

I do not see any indication that this is a HIDAPI issue. And the conversation is stalled. And I don't see how HIDAPI mainteinars can help here.