dasharo-issues icon indicating copy to clipboard operation
dasharo-issues copied to clipboard

ESXI TPM driver fails to load on VP3210

Open mysteriousCzrsX opened this issue 11 months ago • 4 comments

Component

Dasharo firmware

Device

Protectli VP3210, VP2430

Dasharo version

v0.9.0-rc6

Dasharo Tools Suite version

No response

Test case ID

No response

Brief summary

When booting a VP3210 platform with the TPM 2.0 module installed the module is not properly recognized by the OS. Running dmesg | grep tpm produces the following output indicating that the tpm driver failed to load.

2025-01-31T14:44:06.262Z cpu0:262144)Loading tpm.v00...
2025-01-31T14:44:06.263Z cpu0:262144)VisorFSTar: 2610: tpm.v00 for 0x601c bytes
2025-01-31T14:44:06.306Z cpu0:262144)Loading tpmesxup.v00...
2025-01-31T14:44:06.307Z cpu0:262144)VisorFSTar: 2610: tpmesxup.v00 for 0x241c bytes
2025-01-31T14:44:09.273Z [INFO] Partially resolved path: /dev/char/vmkdriver/tpm0
2025-01-31T14:44:09.274Z [INFO] Partially resolved path: /dev/char/vmkdriver/tpm0
2025-01-31T14:44:09.670Z cpu1:262371)Activating Jumpstart plugin tpm.
2025-01-31T14:44:09.670Z cpu0:262380)Applying start on plugin tpm
2025-01-31T14:44:09.673Z cpu0:262380)Loading module tpmdriver ...
2025-01-31T14:44:09.674Z cpu0:262380)Elf: 2129: module tpmdriver has license VMware
2025-01-31T14:44:09.676Z cpu0:262380)Device: 193: Registered driver 'tpmdriver' from 10
2025-01-31T14:44:09.676Z cpu0:262380)Device: 250: Unregistered driver 'tpmdriver' from 10
2025-01-31T14:44:09.678Z cpu0:262380)tpmdriver failed to load.
2025-01-31T14:44:09.678Z cpu0:262380)WARNING: Elf: 3284: Kernel based module load of tpmdriver failed: Failure <Mod_LoadDone failed>
2025-01-31T14:44:09.687Z cpu1:262371)ALERT: Jumpstart plugin tpm activation failed.
2025-01-31T14:44:24.745Z cpu3:262371)Activating Jumpstart plugin tpm-eventlog.
2025-01-31T14:44:24.745Z cpu3:262383)Applying start on plugin tpm-eventlog
2025-01-31T14:44:24.746Z cpu0:262371)Jumpstart plugin tpm-eventlog activated.

How reproducible

100%

How to reproduce

Install a TPM 2.0 module in VP3210 platform and boot ESXI. Log in to the system, enter shell and run dmesg | grep tpm.

Expected behavior

Driver loads correctly and TPM 2.0 is seen by the OS

Actual behavior

Driver fails to load ESXI does not recognise the TPM module.

Screenshots

No response

Additional context

No response

Solutions you've tried

No response

mysteriousCzrsX avatar Feb 04 '25 13:02 mysteriousCzrsX

I guess we going back to testing ESXi (at least installation tests). Not sure if this will be relevant to customers, but let's keep it in mind.

miczyg1 avatar Apr 22 '25 14:04 miczyg1

Same on VP2430.

In the vmkernel.log there's:

VMB_TPM: 158: Locality is not valid, access = 0xff
VMB_TPM: 2294: Failed to request locality 0

ESXi expects the TPM2 ACPI table to point to a valid, reserved MMIO region (typically 0xfed40000), and that region to be accessible and mapped. for the locality protocol to work via MMIO.

Seems like the ACPI table is there, however, the actual MMIO address isn't marked in the system memory map. So ESXi hits 0xff (invalid read) → fails locality 0.

Checked on DTS with the same platform:

# dmesg | grep -i tpm
(...)
ACPI: TPM2 ... COREBOOT
tpm_tis MSFT0101:00: 2.0 TPM (device-id 0x1B, rev-id 22)
(...)

#  cat /proc/iomem | grep -i tpm

It looks like Linux is using the TPM via legacy PNP (LPC) driver path, likely via tpm_tis fallback probing, while the firmware didn't actually reserve the memory-mapped TPM registers in a way ESXi expects (i.e., through ACPI and fed40000 region).

I guess a fix would include making sure coreboot reserves the MMIO region (0xfed40000-0xfed44fff) in the e820 map, The TPM2 ACPI table properly references it, and maybe that the TPM_CRB driver support is enabled, or TPM interface is set to FIFO (not CRB) if that’s more stable.

filipleple avatar Apr 22 '25 14:04 filipleple

VMB_TPM: 158: Locality is not valid, access = 0xff VMB_TPM: 2294: Failed to request locality 0 ESXi expects the TPM2 ACPI table to point to a valid, reserved MMIO region (typically 0xfed40000), and that region to be accessible and mapped. for the locality protocol to work via MMIO.

Locality is something else than MMIO address. Requesting a locality is done by writing to TPM registers. the 0xff means that for some reason the access probably failed (as you noted already). @krystian-hebel may know more about requesting a TPM locality.

Seems like the ACPI table is there, however, the actual MMIO address isn't marked in the system memory map. So ESXi hits 0xff (invalid read) → fails locality 0.

MMIO is not reported in the system memory map passed from the firmware. Only RAM and reserved memory regiosn are. It os up to the OS to reserve or mark the MMIO of the devices. The MMIO regions for devices can be obtained in various ways:

  • by querying device registers (.e.g PCI BARs)
  • by querying ACPI resources

In case of the TPM it is the latter and it is correctly reported in the ACPI SSDT and TPM2 table (you may check the min Linux/DTS by dumping HCL).

It looks like Linux is using the TPM via legacy PNP (LPC) driver path, likely via tpm_tis fallback probing, while the firmware didn't actually reserve the memory-mapped TPM registers in a way ESXi expects (i.e., through ACPI and fed40000 region).

Yes, but it doesn't matter here IMO. TPM2.0 devices have MSFT0101 HID and are no longer PNP.

cat /proc/iomem | grep -i tpm

This won't work, because TPM is shown by its ACPI HID name:

fed40000-fed47fff : PCI Bus 0000:00
  fed40000-fed44fff : MSFT0101:00
    fed40000-fed44fff : MSFT0101:00 MSFT0101:00

I guess a fix would include making sure coreboot reserves the MMIO region (0xfed40000-0xfed44fff) in the e820 map, The TPM2 ACPI table properly references it, and maybe that the TPM_CRB driver support is enabled, or TPM interface is set to FIFO (not CRB) if that’s more stable.

FIFO or CRB interface is just a pure TPM capability. If it talk via CRB interface, FIFO won't work, and vice versa. The driver will check the interface used by TPM and use appropriate functions. Also coreboot actually does reserve it:

[DEBUG]  Writing coreboot table at 0x76492000
[DEBUG]   0. 0000000000000000-0000000000000fff: CONFIGURATION TABLES
[DEBUG]   1. 0000000000001000-000000000009ffff: RAM
[DEBUG]   2. 00000000000a0000-00000000000fffff: RESERVED
[DEBUG]   3. 0000000000100000-0000000076450fff: RAM
[DEBUG]   4. 0000000076451000-000000007652efff: CONFIGURATION TABLES
[DEBUG]   5. 000000007652f000-00000000765d0fff: RAMSTAGE
[DEBUG]   6. 00000000765d1000-0000000076bfffff: CONFIGURATION TABLES
[DEBUG]   7. 0000000076c00000-00000000803fffff: RESERVED
[DEBUG]   8. 00000000c0000000-00000000cfffffff: RESERVED
[DEBUG]   9. 00000000f8000000-00000000f9ffffff: RESERVED
[DEBUG]  10. 00000000fb000000-00000000fb000fff: RESERVED
[DEBUG]  11. 00000000fc800000-00000000fe7fffff: RESERVED
[DEBUG]  12. 00000000feb00000-00000000feb7ffff: RESERVED
[DEBUG]  13. 00000000fec00000-00000000fecfffff: RESERVED
[DEBUG]  14. 00000000fed40000-00000000fed6ffff: RESERVED
[DEBUG]  15. 00000000fed80000-00000000fed87fff: RESERVED
[DEBUG]  16. 00000000fed90000-00000000fed92fff: RESERVED
[DEBUG]  17. 00000000feda0000-00000000feda1fff: RESERVED
[DEBUG]  18. 00000000fedc0000-00000000feddffff: RESERVED
[DEBUG]  19. 0000000100000000-000000047fbfffff: RAM

The unreserved memory area for TPM is rather unlikely cause of the problems. But if you have more logs which indicate otherwise, please share them @filipleple .

Let's try to tackle the locality problem first.

miczyg1 avatar Apr 23 '25 16:04 miczyg1

Locality is not valid, access = 0xff

This looks like what read from unmapped memory would look like, so it is MMIO issue in my opinion. For TIS (FIFO), checking the locality consists of reading an 8-bit register (per locality) from MMIO, and it seems that this is what's failing here. Does this OS have devmem or similar tool for reading memory? If yes, can you read bytes from 0xfed40000, 0xfed41000, 0xfed42000, 0xfed43000 and 0xfed44000?

krystian-hebel avatar Apr 23 '25 16:04 krystian-hebel

Dasharo v0.9.0, Protectli VP3210: Issue still present.

matmacieje avatar Jul 17 '25 11:07 matmacieje

on VP2440 0.9.0-rc4 ESXi recognizes TPM but cannot connect to it

Image

esxidmesg.log

wiktormowinski avatar Aug 27 '25 11:08 wiktormowinski