firmware-lenovo icon indicating copy to clipboard operation
firmware-lenovo copied to clipboard

ThinkPad Universal USB-C Dock , missing FW

Open joakim-tjernlund opened this issue 2 years ago • 46 comments

Just got an ThinkPad Universal USB-C Dock and tried to update FW but there is no FW:

├─ThinkPad Universal USB-C Dock:
│     Device ID:          7a2034b0a5bfbc758ec5bfdce77b0484bdc0b46f
│     Summary:            Dock Management Controller Device
│     Current version:    0.0.0.0
│     Vendor:             Lenovo (USB:0x17EF)
│     Install Duration:   1 minute
│     Serial Number:      1SSD21B41469ZVR08KVN
│     GUIDs:              959b1e72-280e-5453-a5eb-d9d5c7674211 ← USB\VID_17EF&PID_30A9
│                         fa0b1b28-bca0-5577-a670-a7e507f03bda ← USB\VID_17EF&PID_30A9&REV_3073
│     Device Flags:       • Updatable
│                         • System requires external power source
│                         • Device stages updates
│                         • Device can recover flash failures
│                         • Signed Payload

joakim-tjernlund avatar Jun 24 '22 02:06 joakim-tjernlund

The FW is under embargo but should be moving to stable soon. I have tested it myself - but they wanted the Lenovo QA team to check it too. I've asked for an ETA.

As a side Q - which dock do you have? Hoping it's the 40AS or 40AY model....

mrhpearson avatar Jun 24 '22 19:06 mrhpearson

Don't know the exact model(I am remote ATM), I was hoping you could see that from list above? Also, is it not odd that Current version is 0.0.0.0 ?

hmm, lsusb says: Bus 003 Device 017: ID 17ef:30a9 Lenovo 40AY

Using fwupd-1.8.1

joakim-tjernlund avatar Jun 26 '22 13:06 joakim-tjernlund

PXE boot does not seem to work on this dock, just tried remotely to initiate a PXE boot but the computer never came up.

joakim-tjernlund avatar Jun 26 '22 15:06 joakim-tjernlund

40AY is good. That's the firmware that is now online under embargo and waiting on QA. It's a very hectic time of year as all the new platforms are landing so I don't have an ETA yet - but it is coming. For PXE - is that because of MAC passthru by any chance? There's a patch upstream that might be useful - though I note it doesn't cover your dock. Do you know if your dock supports MAC passthru? Wondering if it needs to be added... https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=f01cdcf891a5

mrhpearson avatar Jun 27 '22 17:06 mrhpearson

40AY is good. That's the firmware that is now online under embargo and waiting on QA. It's a very hectic time of year as all the new platforms are landing so I don't have an ETA yet - but it is coming. For PXE - is that because of MAC passthru by any chance? There's a patch upstream that might be useful - though I note it doesn't cover your dock. Do you know if your dock supports MAC passthru? Wondering if it needs to be added... https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=f01cdcf891a5

I have no idea, how to know? I figured it was because of old Dock FW ? Also, that patch has not reached stable, I guess it is because it was not tagged with stable so it never got their attention.

joakim-tjernlund avatar Jun 27 '22 18:06 joakim-tjernlund

Got PXE working now, ipconfig in kernel is slow to complete bringing up USB IP I/F though. Really need that Dock FW now ...

joakim-tjernlund avatar Jun 28 '22 20:06 joakim-tjernlund

@mrhpearson 40AS(Thinkpad usc c dock gen 2) is getting support too? Hopefully DP hub updates are supported as well, I can't even get the DP hub to update on windows. It just times out.

madushan1000 avatar Jul 09 '22 10:07 madushan1000

We still experience some stability problems with the Lenovo 40AY Dock. Fwupdmgr say we have the latest FW: 0.0.4.6 However looking at https://support.lenovo.com/us/en/downloads/ds548947 there is FW 3.0.80 for Windows. The FW versioning numbering does not match Linux and it is impossible to tell what FW Linux really uses. How can I match Linux FW with Windows ?

joakim-tjernlund avatar Feb 20 '23 11:02 joakim-tjernlund

Hi - I've asked the dock team about this. Afraid I don't know the answer. Will let you know when I hear back Mark

mrhpearson avatar Feb 23 '23 02:02 mrhpearson

Hi - they're asking:

  1.  Details on the stability issue - ideally a pic or video if possible so they can understand the details of that
    
  2.  Can you help to confirm the current Dock FW version? Can you get me the output from fwupdmgr get-devices
    
  3.  Any details on how to reproduce; what kernel version & distro are you running
    

Thanks mark

mrhpearson avatar Feb 24 '23 19:02 mrhpearson

  1. No pics/video, sorry. I can describe what we have seen:
  • Loss of Dock eth port, it is often gone after an Linux reboot, replugging Dock USB cable usually restores it

  • When Dock eth work we see alot of this kernel msg: r8152 2-3.1:1.0: load rtl8153b-2 v1 10/23/19 successfully

  • persistent UCSI errors: ucsi_acpi USBC000:00: PPM init failed (-110) That seems to steam from this problem: https://lore.kernel.org/linux-usb/[email protected]/ applying that patch makes ucsi a little happier but still see this: ucsi_acpi USBC000:00: ucsi_handle_connector_change: GET_CONNECTOR_STATUS failed (-110)

  • Rare, but loss off Dock HDMI port, replugging USB cable did not help but reboot of Linux did.

─ThinkPad Universal USB-C Dock:
│ │   Device ID:          7a2034b0a5bfbc758ec5bfdce77b0484bdc0b46f
│ │   Summary:            Dock Management Controller Device
│ │   Current version:    0.0.4.6
│ │   Vendor:             Lenovo (USB:0x17EF)
│ │   Install Duration:   1 minute
│ │   Serial Number:      1SSD21B41469ZVR08KVN
│ │   GUIDs:              959b1e72-280e-5453-a5eb-d9d5c7674211 ← USB\VID_17EF&PID_30A9
│ │                       fa0b1b28-bca0-5577-a670-a7e507f03bda ← USB\VID_17EF&PID_30A9&REV_3073
│ │   Device Flags:       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Device stages updates
│ │                       • Device can recover flash failures
│ │                       • Signed Payload
│ │ 
│ ├─CCG4:
│ │     Device ID:        1f5112276f265b9726e224103cd608c0acb6b026
│ │     Current version:  3.2.1.1652
│ │     Vendor:           Lenovo (USB:0x17EF)
│ │     GUID:             671dec65-a2d3-5899-bb65-678605cdf74f ← USB\VID_17EF&PID_30A9&CID_02
│ │   
│ └─DMC:
│       Device ID:        194de6dea19ac74c18f9013822a06281226c3cfe
│       Current version:  3.3.0.1746
│       Vendor:           Lenovo (USB:0x17EF)
│       GUID:             2f9f0d4e-3713-5086-af85-b8f815549986 ← USB\VID_17EF&PID_30A9&CID_00

kernel 6.1.x, x=10-14 dist: Gentoo I don't think this is distribution specific. More kernel, BIOS FW and Dock FW. I Got a ThinkPad T14 Gen 3 but we see this on all our laptops

joakim-tjernlund avatar Feb 27 '23 09:02 joakim-tjernlund

Thanks - passed on to the FW team. They are looking at updating the FW on LVFS too - there were some previous issues that they are looking into apparently (sorry - I don't have any details yet) Mark

mrhpearson avatar Feb 27 '23 17:02 mrhpearson

Now we are seeing this OOPS also(kernel >= 6.1.14) when unplugging Dock USB:

[ 2286.499615] BUG: kernel NULL pointer dereference, address: 0000000000000398
[ 2286.499618] #PF: supervisor write access in kernel mode
[ 2286.499619] #PF: error_code(0x0002) - not-present page
[ 2286.499620] PGD 0 P4D 0 
[ 2286.499621] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 2286.499622] CPU: 0 PID: 3299 Comm: kworker/0:0 Tainted: G           O       6.1.15-x86_64 #1
[ 2286.499623] Hardware name: LENOVO 21AH00CSMX/21AH00CSMX, BIOS N3MET11W (1.10 ) 12/07/2022
[ 2286.499624] Workqueue: kacpi_notify acpi_os_execute_deferred
[ 2286.499627] RIP: 0010:queue_work_on+0x4/0x30
[ 2286.499630] Code: 82 48 c7 c7 88 97 0d 83 e8 99 5d 5c 00 3b 05 83 86 98 01 89 c1 41 89 c6 0f 82 65 ff ff ff e9 cb fe ff ff 0f 1f 0
0 53 9c 5b fa <f0> 48 0f ba 2a 00 73 0a 31 c0 80 e7 02 74 01 fb 5b c3 e8 75 fb ff
[ 2286.499631] RSP: 0018:ffffc90003cd7e50 EFLAGS: 00010002
[ 2286.499632] RAX: 0000000000000368 RBX: 0000000000000202 RCX: 000000000037c080
[ 2286.499633] RDX: 0000000000000398 RSI: ffff88810004c600 RDI: 0000000000000040
[ 2286.499633] RBP: 0000000000000004 R08: ffff88810d1a4d38 R09: 0000000000000000
[ 2286.499634] R10: 0000000000000000 R11: 000000000000005f R12: ffff88844f42aa40
[ 2286.499634] R13: ffff88844f431d00 R14: 0000000000000000 R15: ffff88844f431d05
[ 2286.499635] FS:  0000000000000000(0000) GS:ffff88844f400000(0000) knlGS:0000000000000000
[ 2286.499635] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2286.499636] CR2: 0000000000000398 CR3: 000000000480a004 CR4: 0000000000770ef0
[ 2286.499637] PKRU: 55555554
[ 2286.499637] Call Trace:
[ 2286.499638]  <TASK>
[ 2286.499640]  ucsi_acpi_notify+0x5f/0x69 [ucsi_acpi]
[ 2286.499645]  acpi_ev_notify_dispatch+0x37/0x4c
[ 2286.499647]  acpi_os_execute_deferred+0xb/0x20
[ 2286.499648]  process_one_work+0x1db/0x390
[ 2286.499649]  worker_thread+0x45/0x3c0
[ 2286.499650]  ? rescuer_thread+0x3a0/0x3a0
[ 2286.499650]  kthread+0xd2/0x100
[ 2286.499652]  ? kthread_complete_and_exit+0x20/0x20
[ 2286.499653]  ret_from_fork+0x1f/0x30
[ 2286.499655]  </TASK>
[ 2286.499655] Modules linked in: rfcomm snd_hrtimer snd_seq nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype dm_thin_pool dm_per
sistent_data dm_bio_prison dm_bufio cmac snd_usb_audio snd_usbmidi_lib snd_rawmidi usbhid snd_seq_device bnep btusb btrtl btbcm btmtk
 btintel uvcvideo videobuf2_vmalloc bluetooth videobuf2_memops videobuf2_v4l2 ecdh_generic ecc videodev videobuf2_common xt_CHECKSUM 
xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle
 iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter ip_tables x_tables snd_ctl_led snd_soc_skl_hda_dsp snd_
soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic vfat fat joy
dev hid_multitouch snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire
_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof
[ 2286.499678]  i2c_designware_platform i2c_designware_core snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match 
snd_soc_acpi soundwire_bus x86_pkg_temp_thermal snd_soc_core snd_compress intel_rapl_msr iwlmvm ac97_bus think_lmi mac80211 firmware_
attributes_class snd_hda_intel wmi_bmof snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm_intel i915 iwlwifi snd_hda_core thinkpa
d_acpi processor_thermal_device_pci intel_lpss_pci snd_hwdep kvm intel_gtt nvram intel_lpss processor_thermal_device snd_pcm ledtrig_
audio irqbypass ghash_clmulni_intel pcspkr efi_pstore ucsi_acpi platform_profile cfg80211 mfd_core i2c_i801 typec_ucsi processor_ther
mal_rfim snd_timer rfkill drm_buddy i2c_smbus roles mei_me processor_thermal_mbox xhci_pci snd processor_thermal_rapl mei intel_rapl_
common xhci_hcd thunderbolt drm_display_helper video typec int3403_thermal int340x_thermal_zone i2c_hid_acpi i2c_hid intel_hid pinctr
l_tigerlake wmi sparse_keymap int3400_thermal pinctrl_intel
[ 2286.499698]  acpi_thermal_rel vboxnetadp(O) vboxnetflt(O) vboxdrv(O) msr fuse dm_mod dax nfsd autofs4
[ 2286.499703] CR2: 0000000000000398
[ 2286.499703] ---[ end trace 0000000000000000 ]---

And the laptop needs a power off to recover.

Any update on Dock FW?

joakim-tjernlund avatar Mar 07 '23 09:03 joakim-tjernlund

I've been looking into that - and it's weird. I see it with the TBT dock FWIW. I don't see the problem with a vanilla build from Linus's tree of the 6.2 kernel - but with the Fedora 6.2 kernel I do. As soon as I add any instrumentation to debug it the issues goes away (though then the IGC kernel driver barfs...).

mrhpearson avatar Mar 08 '23 15:03 mrhpearson

My guess would that you somehow mange to emit some USCI notification from the dock even though UCSI init fails in the kernel and then kernel driver isn't fully setup to handle UCSI notifications.

Double error of sorts, kernel does not properly check all UCSI msgs and does a NULL ptr access when it gets some unexpected UCSI msg.

OOPS happens is the end of ucsi_acpi_notify, maybe ae->complete isn't setup ?

joakim-tjernlund avatar Mar 08 '23 16:03 joakim-tjernlund

I'm not sure - but as a random thing to try if you don't mind...can you rmmod igc before disconnecting the dock?

When I instrument the kernel I get errors in the IGC module where it's unable to read HW (presumably because it's gone). I'm wondering if something in there is trampling or causing issues. At least in my build without the IGC module it's working - but I'm hesitant to point fingers yet...

mrhpearson avatar Mar 10 '23 20:03 mrhpearson

I have blacklisted the ucsi module for now. My kernel does not build igc so there is nothing to remove. Do I need it for some Lenovo machines ?

joakim-tjernlund avatar Mar 11 '23 16:03 joakim-tjernlund

Turns out smarter people than me have figured this out... https://lore.kernel.org/linux-usb/[email protected]/T/#t

Not tested yet - but it should fix the problem. I'll give this a whirl on my system shortly to confirm.

I think the IGC driver is used on the TBT4 dock - but you can ignore it for your case.

Mark

mrhpearson avatar Mar 12 '23 02:03 mrhpearson

That 3 patches looks nice, I will wait until these hit stable though.

joakim-tjernlund avatar Mar 12 '23 11:03 joakim-tjernlund

Would be nice if you could look into these as well;

  • Loss of Dock eth port, it is often gone after an Linux reboot, replugging Dock USB cable usually restores it

  • When Dock eth work we see alot of this kernel msg: r8152 2-3.1:1.0: load rtl8153b-2 v1 10/23/19 successfully

The kernel log is full of this msg.

joakim-tjernlund avatar Mar 12 '23 16:03 joakim-tjernlund

For the original source of this ticket - USB-c dock FW on LVFS.

Team have uploaded the update to 3.0.8.5, but it's currently under embargo. I've just tested it and successfully updated. The Lenovo QA team need to sign off before it will be released to stable so I'm pushing to make that happen

Haven't tested the eth port issues - but I've not seen problems there myself and I am using the Ethernet port successfully (and was previously too)

mrhpearson avatar Mar 17 '23 18:03 mrhpearson

For the original source of this ticket - USB-c dock FW on LVFS.

Team have uploaded the update to 3.0.8.5, but it's currently under embargo. I've just tested it and successfully updated. The Lenovo QA team need to sign off before it will be released to stable so I'm pushing to make that happen

Great ! Should be any day now I hope.

Haven't tested the eth port issues - but I've not seen problems there myself and I am using the Ethernet port successfully (and was previously too)

lost eth I/F still happens from time to time after reboot. the 'load rtl8153b-2 v1 10/23/19 successfully' msg is always there, strange you don't see it. I got MAC pass-through in BIOS, maybe related ? No eth plugged to the Dock.

The UCSI fixes seems stuck in Greg's USB tree for now.

joakim-tjernlund avatar Mar 21 '23 16:03 joakim-tjernlund

@mrhpearson , the UCSI fixes has been released, now would be a good time to lift the embargo on the Dock FW.

joakim-tjernlund avatar Mar 30 '23 15:03 joakim-tjernlund

Ack - it's still under testing and is scheduled to be complete in the next couple of weeks. Mark

mrhpearson avatar Mar 30 '23 17:03 mrhpearson

Hi!

Sorry for barging in but I don't know where else to find info. I just got a brand new Thinkpad E15 Gen4 with a 40AY dock and can't get the displays to work. Keyboard and mouse work fine, so does power. I have a bunch of thunderbolt errors in dmesg and no thunderbolt devices showing up in /sys. I don't know much about fwupd but the dock isn't showing up in there but is showing up with lsusb.

Can you help with things I should check?

OS : Debian 11 Kernel : 5.10.0-21-amd64

Thanks!

Leobaillard avatar Apr 06 '23 16:04 Leobaillard

Can you help with things I should check?

OS : Debian 11 Kernel : 5.10.0-21-amd64

That is way too old kernel to talk to these docks, use latest 6.1.23 (.22 and .23 has usci fixes need)

joakim-tjernlund avatar Apr 06 '23 17:04 joakim-tjernlund

Can you help with things I should check? OS : Debian 11 Kernel : 5.10.0-21-amd64

That is way too old kernel to talk to these docks, use latest 6.1.23 (.22 and .23 has usci fixes need)

Thanks a lot for your answer, that was it! For now, there is only up to 6.1.20 available in the debian unstable repository, but at least I can have both screens working through the dock. I guess I'll wait for a newer release to get the stability improvements.

Leobaillard avatar Apr 06 '23 22:04 Leobaillard

Can you help with things I should check? OS : Debian 11 Kernel : 5.10.0-21-amd64

That is way too old kernel to talk to these docks, use latest 6.1.23 (.22 and .23 has usci fixes need)

Thanks a lot for your answer, that was it! For now, there is only up to 6.1.20 available in the debian unstable repository, but at least I can have both screens working through the dock. I guess I'll wait for a newer release to get the stability improvements.

Meanwhile, don't unplug dock but if you need to, make sure screen is not locked. There are OOPSes in there waiting ...

You can blacklist the ucsi_acpi kernel module should UCSI misbehave, you don't need it for basic operation.

joakim-tjernlund avatar Apr 07 '23 06:04 joakim-tjernlund

For the original source of this ticket - USB-c dock FW on LVFS. Team have uploaded the update to 3.0.8.5, but it's currently under embargo. I've just tested it and successfully updated. The Lenovo QA team need to sign off before it will be released to stable so I'm pushing to make that happen

Great ! Should be any day now I hope.

@mrhpearson , still no new USB dock FW. Some problem ?

joakim-tjernlund avatar Apr 24 '23 12:04 joakim-tjernlund

I pinged them a week ago and the update was still with the QA team and not ready for release yet. There weren't any issues reported so hopefully it's just chewing it's way through the process

mrhpearson avatar Apr 25 '23 00:04 mrhpearson