ape `ImportError: symbol not found` when running `ape test` on macOS

Environment information

OS: macOS
Python Version: 3.10.6
ape and plugin versions:

$ ape --version
0.5.1

$ ape plugins list
Installed Plugins:
  optimism    0.5.0a1
  vyper       0.5.0

Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

$ cat ape-config.yaml
plugins:
  - name: vyper
  - name: optimism

What went wrong?

When running ape test (or pytest), I am getting the following error:

ImportError: dlopen(/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

Full traceback:

❯ ape test
Traceback (most recent call last):
  File "/Users/merc1er/.pyenv/versions/ape/bin/ape", line 8, in <module>
    sys.exit(cli())
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ape/_cli.py", line 40, in invoke
    return super().invoke(ctx)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ape_test/_cli.py", line 17, in cli
    return_code = pytest.main([*pytest_args], ["ape_test"])
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 145, in main
    config = _prepareconfig(args, plugins)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1017, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1305, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1188, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/importlib_metadata/__init__.py", line 203, in load
    module = import_module(match.group('module'))
  File "/Users/merc1er/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/web3/tools/__init__.py", line 1, in <module>
    from .pytest_ethereum import (  # noqa: F401
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/web3/tools/pytest_ethereum/deployer.py", line 12, in <module>
    from ethpm import (
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/__init__.py", line 19, in <module>
    from .package import Package  # noqa: E402, F401
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/package.py", line 47, in <module>
    from ethpm.dependencies import (
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/dependencies.py", line 7, in <module>
    from ethpm.validation.package import (
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/validation/package.py", line 11, in <module>
    from ethpm._utils.ipfs import (
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/_utils/ipfs.py", line 18, in <module>
    from google.protobuf.descriptor import (
  File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: dlopen(/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

This however works fine in GitHub actions (ubuntu-latest).

Sep 22 '22 02:09 merc1er

How to reproduce:

On macOS (M1 chip), in a clean virtualenv:

pip install eth-ape
ape init
ape test

Sep 22 '22 03:09 merc1er

That's a very strange one, @helloibis has an M1 mac and can maybe help debug this

Sep 22 '22 15:09 fubuloubu

@merc1er @fubuloubu yep, able to reproduce this! Going to take a look now

Sep 22 '22 15:09 ghost

@fubuloubu @merc1er seems to be an issue with the web3.py dependency of protobuf, even on the latest web3.py pre-release 6.0.0b5

In the meantime I was able to manually fix the import error by downgrading protobuf by a point version:

(venv) $ pip uninstall protobuf
(venv) $ pip install protobuf==3.20.1

There's an upstream issue at https://github.com/ethereum/web3.py/issues/2654, but I'm not sure how much work getting them on 4.x will be. I run into this issue if I install the latest protobuf rather than downgrading to 3.20.1:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Sep 22 '22 16:09 ghost

3.20.2 was released on Sept. 13th. We could pin to 3.20.1 in Ape. Currently, our web3.py version is pinned anyway, so it wouldn't cause any unexpected problems. Then when we upgrade web3.py to its production release, we can delete the protobuf pin then.

Sep 22 '22 16:09 antazoey

This has been fixed upstream and should be in a release next week

https://github.com/ethereum/web3.py/pull/2657

Once their release is out we can update our dependency!

Sep 22 '22 20:09 ghost

Thanks for the quick turnaround.

I can confirm #1071 works with protobuf==3.20.1 and fixes this for me.

Sep 23 '22 00:09 merc1er

Web3 just released their new beta version, so I've opened up https://github.com/ApeWorX/ape/pull/1082 to see about just updating our dependency. Will close out #1071 once we find out if the new PR suffices 🙂

Sep 26 '22 21:09 ghost

ape ape copied to clipboard

`ImportError: symbol not found` when running `ape test` on macOS

Environment information

What went wrong?

How to reproduce:

ape
ape copied to clipboard