Import py_client fails for not provided libpython3.8.so.1.0

[mh-private]

Hi,

in my project I want to use dm-acme as rl framework. However, when importing e.g. from acme.jax import experiments I get an error for missing shared object file libpython3.8.so.1.0 required from dm-launchpad.

Error message:

File "/home/.../python3_deps_pypi__dm_launchpad/courier/python/client.py", line 30, in <module>
    from courier.python import py_client
ImportError: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

After an internal discussion we concluded that python 3.8 C extensions are not supposed to ask for libpython3.8.so.1.0 (This is also the conclusion in this thread).

So my question is, whether launchpad is built with a shared library python? If this is the case, is this on purpose or could this be changed to be linked with a static one?

[kinalmehta]

Similar issue mentioned in #25

If using conda, I solved it by updating LD_LIBRARY_PATH as follows

mkdir -p $CONDA_PREFIX/etc/conda/activate.d
mkdir -p $CONDA_PREFIX/etc/conda/deactivate.d
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'unset LD_LIBRARY_PATH' > $CONDA_PREFIX/etc/conda/deactivate.d/env_vars.sh

You can run the above after activating your conda environment.

[mh-private]

I am not using conda but we use bazel to build our entire repository. As mentioned, I think if launchpad is built with python 3.8 or higher it should not require the shared library.

Current path I am investigating is providing an empty libpython library. Which seems to be accepted. However, I am running into an issue with protobuf.

ImportError: /home/.../python3_deps_pypi__dm_launchpad/courier/python/libserialization_cc_proto.so: undefined symbol: _ZNK6google8protobuf7Message11GetTypeNameEv

From discussions I found (e.g. https://groups.google.com/g/protobuf/c/6lFyIbbQZVA) it seems to be a problem when not linking against protobuf when compiling the proto.

Any insights here?

[joanna-yoo]

Hi, I am having the same issue has @mh-private - I am also running into the same issue.

    from courier.python import py_client
ImportError: /usr/local/google/home/.../lib/python3.8/site-packages/courier/python/libserialization_cc_proto.so: undefined symbol: _ZNK6google8protobuf7Message11GetTypeNameB5cxx11Ev

If you could look into this, it would be much appreciated!

[mh-private]

So update from my side:

I think the proto issue of undefined symbol, results from launchpad being built against a different version of tensorflow than the one I am using. So probably next week I will try to build against my tf and see whether that resolves the issue and post an update here.

[eugeneteoh]

I had the same problem with the r0.5.2 branch. I figured out that tf version 2.8 causes that error. I upgraded to 2.9 and it worked. Created a PR for this. Master branch is broken though.

[frederikschubert]

I fixed the build for the HEAD~1 commit of the master branch (https://github.com/deepmind/launchpad/commit/221109bada089447733260c28fe9a71e8e233c9d) and tensorflow 2.12.0 here. This works with the latest version of reverb (0.11.0).

[ethanluoyc]

@frederikschubert fix works for me and I can build against TF 2.12.0 and Reverb 0.11.0 successfully for Python 3.9. I put a pre-built wheel for py39 at https://github.com/ethanluoyc/launchpad/releases

Personally, I enjoy using the few RL-related libraries from DM (reverb, envlogger, launchpad). One painful bit is when it doesn't work then building from source is painful, also the maintainers do not seem to be responsive to these things in the sense that even if sometimes we can fix these issues from a fork accepting the changes take a long time.