import magic fails when importing from different packages

[brando90]

I was forced to do

import capnp
# import dag_api_capnp
# print(dag_api_capnp)
capnp.remove_import_hook()
dag_api_capnp = str(Path('~/ML4Coq/ml4coq-proj/data_lib/dag/dag_api.capnp').expanduser())
dag_api_capnp = capnp.load(dag_api_capnp)

In the files that use the scheme because when I try to import a package in a completely different path that uses the dag_api_capnp I get the error that the import doesn't exist:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/brando/ML4Coq/ml4coq-proj/data_lib/dag/dag_dataloader.py", line 13, in <module>
    import dag_api_capnp
ModuleNotFoundError: No module named 'dag_api_capnp'

So this is the challenge:

I run python file main.py that imports uses_capnp.py and that is when it fails unless I put the absolute path to the schema. How can this be fixed?

[haata]

Without a running example it'll be hard to debug this.

However you can see what the import hooks is doing here: https://github.com/capnproto/pycapnp/blob/master/capnp/lib/capnp.pyx#L4114 (add_import_hook).

[robo-wylder]

I'm pretty sure this is due to the fact that when the importer is part of a top level package it will search all the system paths appropriately, however, when it is part of a sub-package, it only searches the sub package. This is what the logic indicates in the pycapnp importer linked below when taken in accordance with the behavior described in the python standard. I think to make this reasonable, the logic should simple prepend the local paths to syspath so they are searched first during all imports for that module. I get this erroneous behavior when importing from a python module, but in a python repl it works as expected.

The pycapnp import logic checks the path argument and subsequently only search that for imports, it should simply prepend that path, see specifically the last 4 lines of this snippet: https://github.com/capnproto/pycapnp/blob/e73261fe847223d24d9d5c2cdd9fdb445bcb4fd9/capnp/lib/capnp.pyx#L4374-L4399

The python import behavior describing when that path parameter is None or __path__