Import problem for diatheke leads to "Module Not Found" Error

[geoffr98]

OS/Platform: Docker Image running Debian GNU/Linux 10 (buster) python -m nuitka --version:

0.6.7
Python: 3.7.5 (default, Nov 23 2019, 06:10:46)
Executable: /usr/local/bin/python
OS: Linux
Arch: x86_64

nuitka installed with: pip3 install nuitka

Simplified python file showing issue is just one line: import diatheke

diatheke module was installed as using a requirements file with the following line: git+git://github.com/cobaltspeech/sdk-diatheke#egg=cobalt-diatheke&subdirectory=grpc/py-diatheke

nuitka build was done with the following (no errors given): python -m nuitka --plugin-enable=pylint-warnings --recurse-all --standalone --follow-imports --include-plugin-directory=/usr/local/lib/python3.7/site-packages/grpc --include-plugin-directory=/usr/local/lib/python3.7/site-packages/diatheke test.py

When I run the code I get a ModuelNotFound Error (no errors when running python test.py):

# test.dist/test
Traceback (most recent call last):
  File "/app/test.dist/test.py", line 2, in <module>
  File "/app/test.dist/diatheke/__init__.py", line 25, in <module diatheke>
  File "/app/test.dist/diatheke/client.py", line 19, in <module diatheke.client>
ModuleNotFoundError: No module named 'diatheke_pb2_grpc'

Any ideas as to why I'm getting this error?

[kayhayen]

I managed to reproduce it.

[kayhayen]

I believe, I was wrong with my initial assumption, this seems to be a normal import failing, for which I don't have an explanation yet, probably something about from imports not being relative enough inside a package.

[kayhayen]

This is really, really strange. When trying --recurse-all, I am getting this error:

Traceback (most recent call last): File "/home/hayen/test/diatheke/Mini.py", line 1, in import lib.hints File "/home/hayen/test/diatheke/lib/python3.7/site-packages/diatheke/init.py", line 25, in from .client import Client File "/home/hayen/test/diatheke/lib/python3.7/site-packages/diatheke/client.py", line 17, in import grpc File "/home/hayen/test/diatheke/lib/python3.7/site-packages/grpc/init.py", line 23, in from grpc._cython import cygrpc as _cygrpc ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/home/hayen/test/diatheke/lib/python3.7/site-packages/grpc/_cython/init.py)

And then, the like that is supposed to contain the name, does not, but there is the extension module that of course should: cygrpc.cpython-37m-x86_64-linux-gnu.so, but this fails to load. This might be the same kind of error though. Maybe the name import to module import logic doesn't treat packages right. Both diatheke and grpc._cython are happening in packages.

[kayhayen]

So, forever it seems, Nuitka has registered its meta path based loader, at the third spot in the meta path based loaders. I forgot precisely why, but it had to do with frozen modules to still be loaded with that importer, and Nuitka not handling that.

The issue seems to be that Nuitka claims to be responsible, while saying it is not, but for accelerated mode, this will only be a partial statement, and later loaders still need to get their turn.

However, there is a new one, for virtualenv's it seems, and Nuitka is now after the __path__ based loader, that is now a separate one. I think this massively changed. Adding Nuitka's loader to the last spot seems to work, but might mean it's not used, but instead the original Python code is.

[kayhayen]

So, it's not so much about order, but about the PatchFinder not knowing about the parent module yet, and its __loader__ being responsible.

There is code in Nuitka for when after a module returns (successfully), that will register it it in sys.path_importer_cache and that has something to do with pkgutil.walk_packages and the like, getting it to work. However, that's not yet effective.

So durting run time of grpc._cython the PathFinder meta path based loader will not know about the responsibility of it for its __path__, and therefore not find the extension module. Shifting that kind of code, to the inside of the module, ought to work, but is of course always a bit tricks. I shall see.

This seems important for compatibility.

[kayhayen]

So I shifted the trick, but no success. The __path__ and sys.path_importer_cache now list the loader, but the later seems unused for import purposes.

I will need to trace the importlib PathFinder working, to see if it outright rejects it, the only thing that I can see, is how the spec value may not be perfect, as it has a __path__ equivalent too.

[kayhayen]

Ok, it's absolutely clear, I monkey patched the importlib and saw that it definitely wants to ask the package loader, which is us, to load anything below its path now. Yet, we don't do it, and then the normal finder claims to not be responsible.

So this will take us to replicate the spec production part, where we check files for existance, and produce the proper spec. Since we do all of this in C, it's going to be a bit annoying to iterate over the list of extensions, etc. but of course it's doable. For a start, we need to do this for extension modules, but source files that were excluded in acceleration mode, I suspect, are also not going to be loaded from source, although that makes sense. Going to test that.

[LeonardoRick]

I don't understand a lot about what's happening but I think I'm facing a similar issue:

Traceback (most recent call last): File "/home/leonardo/iq/iqbots/base/restrict/iq-copy-list-robot-realtime/main.dist/pkgutil.py", line 637, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/home/leonardo/iq/iqbots/base/restrict/iq-copy-list-robot-realtime/main.dist/grpc/_cython/_credentials/roots.pem' Exception ignored in: 'grpc._cython.cygrpc.ssl_roots_override_callback' Traceback (most recent call last): File "/home/leonardo/iq/iqbots/base/restrict/iq-copy-list-robot-realtime/main.dist/pkgutil.py", line 637, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/home/leonardo/iq/iqbots/base/restrict/iq-copy-list-robot-realtime/main.dist/grpc/_cython/_credentials/roots.pem' E0209 16:59:22.808823525 66088 ssl_utils.cc:550] assertion failed: pem_root_certs != nullptr

Any solutions so far? The program starts to run but at the beggining of execution it throws this error.

Thanks a lot.

[kayhayen]

This is part of the current hotfix release, turns out that do sys.path extension with names that should actually be in the module, and relative imports were not always followed, with the combination of thise, this is solved now.