[BUG]: bindings that work from Python fail with "Cannot find class 'py.pybind11_builtins.pybind11_type'" when used from Matlab Python interface

[slayoo]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

Basic things like instantiation of a pybin11-exposed C++ class, that do work when called from Python, stop to work when called from the Matlab Python interface (through the same install pybind11-generated Python package). There is just a "Cannot find class 'py.pybind11_builtins.pybind11_type" error shown. Any clues on how to debug it or what is causing the error? Thanks!

Reproducible example code

Minimal reproducer with succeeding code/GH action in Python and failing GH action for Matlab here: https://github.com/slayoo/pybind_matlab_issue

OK Python: https://github.com/slayoo/pybind_matlab_issue/actions/runs/6911677790/job/18806501098 KO Matlab: https://github.com/slayoo/pybind_matlab_issue/actions/runs/6911677788/job/18806501100

[slayoo]

Fort the record, noting down here the reply from Gitter by @Holt59:

You should report this on the Matlab side because it's most likely not a pybind11 issue. That py. prefix in the class not found is probably the issue, but it's probably added by Matlab or somewhere else because it should not be there.

@acampbel could you help us here? The issue is that while using a C++-implemented package that works OK from Python, when loading it from Matlab, there is some problem loading share libraries as the symbols seem to have an additional "py." prefix that prevents them from being correctly located. The error message is: "Cannot find class 'py.pybind11_builtins.pybind11_type'." (while the class name is just pybind11_builtins.pybind11_type, without the py.). I've prepared a minimal reproducer using the Matlab actions (see links above). Thanks!

[slayoo]

For the record, noting here down a comment from Gitter by @jeandet

could they have done something like this https://stackoverflow.com/questions/6940384/how-to-deal-with-symbol-collisions-between-statically-linked-libraries/6940389#6940389? did you try to do some nm or readelf on their elf files?

[slayoo]

see also here: https://www.mathworks.com/matlabcentral/answers/2031219-matlab-fails-to-run-python-built-with-pybind

[slayoo]

@henryiii, @rwgk, let me take the liberty of "pinging" if you'd have any hints on how to workaround this issue, i.e. make pybind11-based packages usable under the Matlab Python interface?

The issue is (with a minimalistic CI reproducer here: https://github.com/slayoo/pybind_matlab_issue - see also links above to logs of CI runs depicting the problem) that a correctly functioning pybind11-based Python module, when imported in Matlab throws:

  Cannot find class 'py.pybind11_builtins.pybind11_type'.

Implying that Matlab seems to prefix the symbols with the bogus "py.". Thanks!

[rwgk]

if you'd have any hints on how to workaround this issue

I briefly looked, but sorry, I don't have any hints.

The logs will expire, I recommend you paste the critical messages here. It'll also make it more likely that others with the same problem find this bug (and then maybe you can stick your heads together to solve it).

It's not certain (at all) that the problem is rooted in pybind11. If you reduce the problem to a pybind11 PR that fails in the pybind11 GHA I'll take another look.

/home/runner/work/_actions/matlab-actions/run-command/v0/dist/bin/run_matlab_command.sh cd('/tmp/run_matlab_command-E64Swj'); command_a13eb2e9_a96f_46a8_92e6_7cd68565fbda
  
  	Sponsored License -- for use in support of a program or activity sponsored by MathWorks.
  	Not for government, commercial or other non-sponsored organizational use.
  
  
  ver =
  
      '0.1.dev1+gf22d8c9'
  
  	linux-vdso.so.1 (0x00007ffd51ddc000)
  	libstdc++.so.6 => /usr/local/MATLAB/R2022a/sys/os/glnxa64/libstdc++.so.6 (0x00007fe4cc3c2000)
  	libgcc_s.so.1 => /usr/local/MATLAB/R2022a/sys/os/glnxa64/libgcc_s.so.1 (0x00007fe4cc3a8000)
  	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe4cc1a6000)
  	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe4cc057000)
  	/lib64/ld-linux-x86-64.so.2 (0x00007fe4cc5ba000)
  
  ans =
  
       0
  
  {Error using readme
  Cannot find class 'py.pybind11_builtins.pybind11_type'.
  
  Error in command_a[13](https://github.com/slayoo/pybind_matlab_issue/actions/runs/6911677788/job/18806501100#step:7:15)eb2e9_a96f_46a8_92e6_7cd68565fbda (line 1)
  cd('/home/runner/work/pybind_matlab_issue/pybind_matlab_issue'); readme
  }

[slayoo]

Thank you, @rwgk!

If you reduce the problem to a pybind11 PR that fails in the pybind11 GHA I'll take another look.

voilà: https://github.com/pybind/cmake_example/pull/164

[slayoo]

For the record, a workaround for this problem was proposed by @rwgk at https://github.com/pybind/cmake_example/pull/164