tpm2_ptool - Undefined sybol: TSS2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

[Gandalf1783]

Hey, I wanted to use my TPM2 module for storing ssh keys, but I received an error from the tpm2-tools suite.

I ran "tpm2_ptool --help" and received the following stacktrace:

Traceback (most recent call last):
  File "/usr/bin/tpm2_ptool", line 5, in <module>
    from tpm2_pkcs11.tpm2_ptool import main
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/tpm2_ptool.py", line 6, in <module>
    from .commandlets_store import InitCommand  # pylint: disable=unused-import # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/commandlets_store.py", line 13, in <module>
    from .utils import bytes_to_file
  File "/usr/lib/python3.11/site-packages/tpm2_pkcs11/utils.py", line 21, in <module>
    from tpm2_pytss.ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/__init__.py", line 2, in <module>
    from .ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/ESAPI.py", line 2, in <module>
    from .types import *
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/types.py", line 12, in <module>
    from ._libtpm2_pytss import ffi, lib
ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

I added myself to the tss group and running the command using root does nothing else. However, I did ran tpm2_clear before, tho I suspect that it is unrelated to the issue.

[retpolanne]

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal seems to exist on _libtpm2_pytss.abi3.so :thinking:

I've uninstalled tpm2-pkcs11 and let only python-tpm2-pytss 2.1.0-1 installed on Arch and it seems that the error continues showing up even on the 2.1.0-1 version:

>>> from tpm2_pytss.ESAPI import ESAPI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/__init__.py", line 2, in <module>
    from .ESAPI import ESAPI
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/ESAPI.py", line 2, in <module>
    from .types import *
  File "/usr/lib/python3.11/site-packages/tpm2_pytss/types.py", line 12, in <module>
    from ._libtpm2_pytss import ffi, lib
ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

edit: my bad, it has a U in front of it, which means undefined

Seems related to https://github.com/tpm2-software/tpm2-pytss/issues/496

workaround

Installing from pip fixes it :)

python3 -m pip install tpm2-pytss

However, it seems that, at least on Arch, the tpm2-pkcs11 package bundles the python one, so it overwrites it and breaks.

Fixing on Arch (at least)

Rebuilding python-tpm2-pytss seems to have fixed it.

sudo pacman -Rsn tpm2-pkcs11 python-tpm2-pytss
curl -sS https://gitlab.archlinux.org/archlinux/packaging/packages/tpm2-pkcs11/-/raw/main/keys/pgp/5B482B8E3E19DA7C978E1D016DE2E9078E1F50C1.asc\?inline\=false | gpg --import
git clone https://gitlab.archlinux.org/archlinux/packaging/packages/python-tpm2-pytss.git
cd python-tpm2-pytss
makepkg -si
sudo pacman -S tpm2-pkcs11

[JuergenReppSIT]

@whooo can you give advice on how to solve this problem?

[whooo]

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal where removed in tpm2-tss 4.0.0, so it looks like the tpm2-pytss package was built with an older release of tpm2-tss and then the tpm2-tss libraries where upgraded, can you check which version of tpm2-tss you have installed on your systems? How did you install tpm2-pytss?

[williamcroberts]

Wrong project, moving it over to tpm2-pkcs11

[retpolanne]

@whooo I did install mine from the tpm2-tss package on Arch. I don't recall the version but I can check when I reinstall it.

[Gandalf1783]

Sorry, but I cannot test against this issue currently. On Arch it "just works"?!

I had the problems on Fedora, but the distro didnt work out.

But I guess @retpolanne did find a possible solution or workaround :)

[williamcroberts]

Also make sure you don't have multiple .so's on the system. I've seen this happens where it builds against the correct headers but links to a library that wasn't expected. ldd on the shared object will give you where its resolving all the dependencies.

[paranormal]

Hi, I can confirm the issuer is present on Arch as of now.

ImportError: /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so: undefined symbol: Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal

Searching for leftovers with pacreport --unowned-files on affected system would show none of them.

ldd is fine too

~: ldd /usr/lib/python3.11/site-packages/tpm2_pytss/_libtpm2_pytss.abi3.so
	linux-vdso.so.1 (0x00007ffc4ffef000)
	libtss2-esys.so.0 => /usr/lib/libtss2-esys.so.0 (0x00007f92bce19000)
	libtss2-tctildr.so.0 => /usr/lib/libtss2-tctildr.so.0 (0x00007f92bce0f000)
	libtss2-rc.so.0 => /usr/lib/libtss2-rc.so.0 (0x00007f92bce05000)
	libtss2-mu.so.0 => /usr/lib/libtss2-mu.so.0 (0x00007f92bcdb5000)
	libtss2-fapi.so.1 => /usr/lib/libtss2-fapi.so.1 (0x00007f92bccb3000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f92bcac9000)
	libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x00007f92bc5ca000)
	libtss2-sys.so.1 => /usr/lib/libtss2-sys.so.1 (0x00007f92bc5a3000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f92bcff2000)
	libjson-c.so.5 => /usr/lib/libjson-c.so.5 (0x00007f92bc590000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007f92bc4e0000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f92bc4d7000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x00007f92bc4aa000)
	libidn2.so.0 => /usr/lib/libidn2.so.0 (0x00007f92bc488000)
	libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007f92bc446000)
	libpsl.so.5 => /usr/lib/libpsl.so.5 (0x00007f92bc432000)
	libssl.so.3 => /usr/lib/libssl.so.3 (0x00007f92bc392000)
	libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00007f92bc33e000)
	libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007f92bc269000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f92bc25b000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f92bc241000)
	libunistring.so.5 => /usr/lib/libunistring.so.5 (0x00007f92bc087000)
	libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00007f92bbfaf000)
	libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00007f92bbf7f000)
	libcom_err.so.2 => /usr/lib/libcom_err.so.2 (0x00007f92bbf79000)
	libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00007f92bbf6b000)
	libkeyutils.so.1 => /usr/lib/libkeyutils.so.1 (0x00007f92bbf64000)
	libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f92bbf52000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f92bbf2f000)

I could only surmise that the current Arch package is to blame according to what I read above.

[whooo]

I can not speak for the issue with Fedora (but I suspect the same). the tpm2-pytss package gets built against tpm2-tss 3.2.x, later the tpm2-tss package is upgraded to a newer version (4.0.x) which has dropped those two functions as they are deprecated (and nothing really used it). But due to how the tpm2-pytss module is built it's linked against basically all symbols defined in any of the headers.

So no package is really to blame, rebuilding the package in Arch should be enough (and might include some new extras then as well).

@diabonas, do you have any insight on the Arch parts?

[dvzrv]

Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal where removed in tpm2-tss 4.0.0

tpm2-tss has indeed been updated to 4.0.1 (https://gitlab.archlinux.org/archlinux/packaging/packages/tpm2-tss/-/commit/a6479bce838a3a653495704a0bd4419ac4ff6e4d) after updating python-tpm2-pytss to 2.1.0 (https://gitlab.archlinux.org/archlinux/packaging/packages/python-tpm2-pytss/-/commit/9ef963c16c86b4e0ee8b13735c245178531a23dd).

Not sure what exactly happened there, but did we miss a soname change, or was none introduced? (maybe @arojas remembers)

@diabonas, do you have any insight on the Arch parts?

Since he is M.I.A. I have rebuilt our python-tpm2-pytss package.

[dvzrv]

To add further information on the tpm2-tss upgrade (3 -> 4): There are no soname changes introduced, yet @whooo mentioned the removal of Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Marshal and Tss2_MU_TPMS_ALGORITHM_DESCRIPTION_Unmarshal in https://github.com/tpm2-software/tpm2-pkcs11/issues/840#issuecomment-1588136865. Removal is a breaking change, which not only should increase the major version of the project (see semver), but lead to a soname change in the affected library.

For reference: Using soname changes downstreams are able to detect ABI changes, which then lead to rebuilds of all affected consumers of a library. If there is no soname change, then there is no way of knowing if the ABI has changed (and things break, as they did here). As tpm2-tss is far past a stable 1.0.0 release, this needs to be considered carefully and soname changes should be introduced whenever there is a backwards incompatible change (e.g. removal).

The below provides the repod-file output for tpm2-tss 3.2.0-3 and 4.0.1-1. As evidenced by the provides list, there has been no soname change.

repod-file package inspect -Pp /var/cache/pacman/pkg/tpm2-tss-3.2.0-3-x86_64.pkg.tar.zst
{
  "arch": "x86_64",
  "backup": [
    "etc/tpm2-tss/fapi-config.json",
    "etc/tpm2-tss/fapi-profiles/P_ECCP256SHA256.json",
    "etc/tpm2-tss/fapi-profiles/P_RSA2048SHA256.json"
  ],
  "base": "tpm2-tss",
  "builddate": 1667335578,
  "checkdepends": [
    "iproute2",
    "swtpm",
    "uthash"
  ],
  "conflicts": null,
  "depends": [
    "curl",
    "json-c",
    "openssl",
    "libjson-c.so=5-64"
  ],
  "desc": "Implementation of the TCG Trusted Platform Module 2.0 Software Stack (TSS2)",
  "fakeroot_version": "1.29",
  "groups": null,
  "isize": 2999883,
  "license": [
    "BSD"
  ],
  "makedepends": [
    "cmocka",
    "doxygen",
    "libtpms"
  ],
  "makepkg_version": "6.0.2",
  "name": "tpm2-tss",
  "optdepends": null,
  "packager": "Felix Yan <[email protected]>",
  "provides": [
    "libtss2-esys.so=0-64",
    "libtss2-fapi.so=1-64",
    "libtss2-mu.so=0-64",
    "libtss2-rc.so=0-64",
    "libtss2-sys.so=1-64",
    "libtss2-tctildr.so=0-64"
  ],
  "replaces": null,
  "schema_version": 2,
  "url": "https://github.com/tpm2-software/tpm2-tss",
  "version": "3.2.0-3",
  "xdata": []
}

repod-file package inspect -Pp /var/cache/pacman/pkg/tpm2-tss-4.0.1-1-x86_64.pkg.tar.zst
{
  "arch": "x86_64",
  "backup": [
    "etc/tpm2-tss/fapi-config.json",
    "etc/tpm2-tss/fapi-profiles/P_ECCP256SHA256.json",
    "etc/tpm2-tss/fapi-profiles/P_RSA2048SHA256.json"
  ],
  "base": "tpm2-tss",
  "builddate": 1683452210,
  "checkdepends": [
    "iproute2",
    "swtpm",
    "uthash"
  ],
  "conflicts": null,
  "depends": [
    "curl",
    "json-c",
    "openssl",
    "libjson-c.so=5-64"
  ],
  "desc": "Implementation of the TCG Trusted Platform Module 2.0 Software Stack (TSS2)",
  "fakeroot_version": "1.31",
  "groups": null,
  "isize": 3783221,
  "license": [
    "BSD"
  ],
  "makedepends": [
    "cmocka",
    "doxygen",
    "libtpms"
  ],
  "makepkg_version": "6.0.2",
  "name": "tpm2-tss",
  "optdepends": null,
  "packager": "Antonio Rojas <[email protected]>",
  "provides": [
    "libtss2-esys.so=0-64",
    "libtss2-fapi.so=1-64",
    "libtss2-mu.so=0-64",
    "libtss2-rc.so=0-64",
    "libtss2-sys.so=1-64",
    "libtss2-tctildr.so=0-64"
  ],
  "replaces": null,
  "schema_version": 2,
  "url": "https://github.com/tpm2-software/tpm2-tss",
  "version": "4.0.1-1",
  "xdata": []
}

[Gandalf1783]

I didn't have time to test the latest changes yet. I will take a look if it works, but I assume it does.

Thank you very much!