Windows 10 support

[ljani]

FYI, there seems to be some magic dance involved with Windows 10 and Concordance:

• When I first plug in my Harmony 885, Windows gives me a prompt a prompt saying:

Setting up device
A device is undergoing additional setup.

• I wait a little bit and get this:

Device setup complete
Device 'Harmony Remote 0-4.4.2' has completed setup.

• Now concordance.exe -v -i works fine
• But if I try to read the config (concordance.exe -v -c test.EZHex), concordance just freezes until I unplug my remote when it fails with:

ERROR: failed to requesting identity
Failed with error 1

• Now if I replug my remote or had exited concordance with ^C, even the -i command won't work anymore, nothing just happens
• To make the device work again, I need to uninstall the remote (USB Input Device and/or HID-compliant vendor-defined device with VID_046D&PID_C110) from the Device Manager and everything goes to the initial state before the Setting up device prompt
• Now when I replug or "scan for hardware changes" and try to read the config or flash anything right after my remote says USB CONNECTED and Windows has not configured the device yet, everything works including flashing
  • As the connectivity test and update cycle is a bit slow, I need to plug the device in, run the connectivity test, uninstall the device, scan for hardware changes, run the update, but it does work

So, it seems there's something funky going on with Windows and USB.

Thank you for your work!

[swt2c]

I will try to take a look. I only have Windows 10 in a VM, though.

Did you build concordance yourself, or are you using the provided binaries?

[ljani]

I'm using the provided 1.2 binary from SourceForge.

[swt2c]

I can't reproduce the issue. I tried a couple of remotes and I was able to run -i and dump configs repeatedly. Well, with one of the remotes there was an issue writing the config to disk, but that doesn't seem USB related.

The only thing I can think of is that there is a problem with hidapi on Windows 10. They have made one fix in that area. I built a new copy of hidapi from latest git - do you mind trying it? Just unzip and copy this files to c:\Program Files (x86)\concordance and try again.

libhidapi-0.dll.zip

[ljani]

Thanks for looking into the issue.

The newer libhidapi-0.dll does not seem to have any effect.

In fact I noticed that the Windows driver dialog doesn't seem to play any role at all. The problem seems to be related to time(outs?). If I keep spamming concordance.exe -i, the remote responds just fine every time. If I won't do anything for 15 seconds, the next concordance.exe -i call hangs (Run concordance.exe -i once, wait 15s and run concordance.exe -i again). Can you reproduce this?

[swt2c]

No, I cannot reproduce that. That is starting to sound maybe like a USB power management issue. Can you check and see if you can turn off USB power management for the Harmony device and see if that changes things?

[ljani]

No dice, it still behaves as previously. I turned off the "Allow the computer to turn off this device to save power" for the "USB Input Device". The "HID-compliant vendor-defined device" does not seem to have a power management tab. As far as I know, there are no other entries related to the remote in device manager.

[ljani]

I tested the remote on another laptop (Windows 10 x64 version 1703 build 15063.483 also), two different cables and another remote (the same model, Harmony 885) and they all yield the same results.

I also tested the remote in a VM, Windows 10 x86 version 1703 build 15063.483 in VirtualBox 5.1.22 on the very same Windows 10 host where the lockup occurs.. And it works without any hiccups. I also downloaded Windows 10 x64 version 1703 build 15063.0 from here and it seems to work fine also.

So, the issue seems to exhibit only on bare metal Windows 10 installation. Any thoughts?

[swt2c]

Unfortunately, I don't have a bare metal Windows 10 machine to try to reproduce myself.

Can you run a debug build of concordance and provide the output from that?

[ljani]

Sorry for late reply, I was travelling.

Sure, do you have prebuilt binaries somewhere or do I need to setup an environment? I'd prefer the former if possible.

[swt2c]

Yeah, I can build you some debug binaries.

[swt2c]

Here's an installer for a debug version: concordance-installer.exe.zip

[ljani]

Thanks.

Okay, here's an output I got when I ran concordance instantly after connecting the remote:

concordance ❯❯❯ .\concordance.exe -i -v
Concordance 1.2
Copyright 2007 Kevin Timmerman and Phil Dibowitz
This software is distributed under the GPLv3.

DEBUG (FindRemote): Testing: 046D, C245
DEBUG (FindRemote): Testing: 0951, 16A4
DEBUG (FindRemote): Testing: 0951, 16A4
DEBUG (FindRemote): Testing: 046D, C245
DEBUG (FindRemote): Testing: 046D, C110
DEBUG (FindRemote): Found a Harmony!
Requesting identity:  100%             done
  Model: Logitech Harmony 885 (Espresso)
  Skin: 17
  Firmware Version: 4.4
  Firmware Type: 0
  Hardware Version: 2.1.0
  External Flash: 2 MiB - 01:49 AMD Am29LV160BB
  Architecture: 8
  Protocol: 8
  Manufacturer: Harmony Remote 0-4.4.2
  Product: Harmony Remote 0-4.4.2
  IRL, ORL, FRL: 0, 0, 0
  USB VID: 046D
  USB PID: C110
  USB Ver: 0811
  Serial Number: {EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE}
        {8D166CA7-8580-4EB3-837E-41FDBD902335}
        {00619A30-EA3A-451E-A697-19EAD9B45357}
  Config Flash Used: 26% (513 of 1920 KiB)

Success!

And here's one after letting 15 seconds pass:

concordance ❯❯❯ .\concordance.exe -i -v
Concordance 1.2
Copyright 2007 Kevin Timmerman and Phil Dibowitz
This software is distributed under the GPLv3.

DEBUG (FindRemote): Testing: 046D, C245
DEBUG (FindRemote): Testing: 0951, 16A4
DEBUG (FindRemote): Testing: 0951, 16A4
DEBUG (FindRemote): Testing: 046D, C245
DEBUG (FindRemote): Testing: 046D, C110
DEBUG (FindRemote): Found a Harmony!

(Ctrl-C at this point)

Doesn't seem to be much help, are there any hidden switches I should toggle or how else I could help more?

[swt2c]

Yeah, I was hoping that would be more helpful. Unfortunately, it doesn't help much.

I'm a bit at a loss at the moment. My next step would probably be to put a USB analyzer in place and take a look at the packets. I don't suppose you have a USB analyzer, do you? :-) You could try some of the software-based USB capture tools for Windows. Unfortunately, they are not very good, or weren't at least the last time I tried them.

Also, have you tried the official software? Does it have any sort of problems on this system?

[ljani]

Good point about testing the official application. I have some mixed results with it:

0. I installed the official software prior any of these tests
1. Plugging in the remote and letting the theorized timeout expire and only after then launching the official software, it won't find the remote either
2. Plugging in the remote and launching the official software soonish after, the software won't find the remote by the time I get to the "update remote" screen. My computer is a bit old, might be a factor here.
3. Starting the official software and plugging in the remote, Concordance works without any problems even after minutes of inactivity (and so does the official software obviously)
4. I fully exited the official software and kept the USB plugged in, Concordance stopped working as previously after 15 seconds.

I seem to be able to capture USB traffic using Wireshark and USBPcap, but I didn't notice any keep-alives when the official software is running. I'm a little puzzled, it seems that the original software does something, but not keep-alives. I also wonder why the VM works without the official software.

I'll send you some USB captures along with their descriptions to you over email I found from your recent commits in a bit as I do not want to post them here publicly if they contain any personal information. You should be able to open them on any OS with Wireshark 2. I think the email is correct, but just in case please let me know if you didn't receive them.

Now there's at least a little better workaround than just being quick. Feel free to close this issue when you think we've hit a dead end.

Thank you for taking your time with this!

PS. The official application does sit in the taskbar when closed if I do not exit it explicity from the taskbar. PPS. As you can probably see, I have no idea how USB works, hehe.

[swt2c]

I got the captures - thanks! They are interesting. Can you please perform one more capture without the official software running where you perform two sequential runs of concordance -iv but with a short enough wait in between so that the timeout does not occur?

[ljani]

Sure thing! Let me capture one right now and I'll email you it in a bit.

[swt2c]

Thanks for the new capture. So a few thoughts:

1. At the beginning of the set of requests that eventually fails, there is a USBHID SET_IDLE request sent. I don't really understand what this request does, but it sounds suspicious. This isn't something that concordance is sending, and I don't think it is something that hidapi sends either. It must be being sent by the HID driver?
2. I know we explored power management before, but did you try disabling USB selective suspend? Here's a website that talks about how to disable it: https://www.windowscentral.com/how-prevent-windows-10-turning-usb-devices
3. If that doesn't work, we need to somehow figure out what the official software is doing to keep the remote from going idle. Can you do one more capture where you a) start the capture b) start the official software running c) plug in the remote d) run concordance -iv once and e) wait for longer than the idle timeout and run concordance -iv once again.

[ljani]

1. Sorry, no idea
2. USB Selective Suspend didn't cross my mind. However I just tried disabling it and the other power management options in Device Manager and after rebooting the system Concordance still hangs.
3. I think the first capture I sent you ~already illustrates this~ was missing the step d), but I'll send another one with the power management options disabled if that gets rid of the idle request noise.

[swt2c]

Sure enough, when the official software is running, that SET_IDLE request doesn't seem to come. So, it seems like that must be relevant. I don't know whether that SET_IDLE request causes the device to go offline, or whether that's just a side effect of something else that's happening that causes the device to go offline.

[ljani]

I see.

If I understand correctly, initially SET_IDLE is issued always when the remote is plugged in, but after a second or so, the official software does something, which will prevent future SET_IDLE requests (not sure if packets 16-23 in both of the official software captures are related). I think this means that it's only possible to fix Concordance so that you still need to be quick to run the first command?

One way to ease this would be to add a --wait flag to Concordance so that Concordance would idle and detect the remote when it is plugged in. With this at least the subsequent requests could work, if it doesn't require keeping Concordance running. However this won't solve the problem that Concordance still needs to be ran before connecting the remote.

Another interesting fact seems to be that if I close the official software and run Concordance after a while, USBHID seems to send SET_IDLE as the first request only then.

Anyway, I'll be travelling for a week or so, talk to you later. Thank you for taking your time with this!

EDIT: the real fix is of course that there is no --wait flag and everything just works, but I'm not sure if that can be achieved. If only there was a magic WAKE_UP command.