TypeError: _dp_init_subclass() missing 1 required positional argument: 'sub_cls'

[joeyballentine]

Nuitka version: latest stable (0.6.17) & dev (0.6.18rc1) Python version: 3.9.3 Platform: Windows

How did you install Nuitka and Python: tried pip and windows installer

Full traceback:

  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\run.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\nodes\pytorch_nodes.py", line 5, in <module nodes.pytorch_nodes>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\torch\__init__.py", line 705, in <module torch>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\torch\utils\data\__init__.py", line 4, in <module torch.utils.data>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\torch\utils\data\dataset.py", line 266, in <module torch.utils.data.dataset>
  File "C:\Users\Joey\DOCUME~1\Git\project\backend\RUN~1.DIS\torch\utils\data\_typing.py", line 273, in __new__
  File "abc.py", line 85, in __new__
TypeError: _dp_init_subclass() missing 1 required positional argument: 'sub_cls'

I get this when trying to hit my compiled API. It appears to be an issue with PyTorch? Clearly it's something to do with the abstract base class. I can't really post example code as I'd have to post a ton of it.

I've tried running this in --onefile mode and --standalone mode. I tried the stable and development versions of Nuitka. Nothing seems to make this work. It works perfectly fine running it regularly with python but compiled I get this error. It compiles perfectly fine, and it runs, but as soon as it starts running a file that imports PyTorch I'm getting this.

Sorry if this is not enough information, I can provide more specific info if needed.

[kayhayen]

That's interesting, it probably is indeed a compatibility problem with PyTorch. I have to admit, support for torch is currently not great, but it will arrive in the coming months.

[kayhayen]

Can you confirm that "import torch" reproduces this for you?

[joeyballentine]

Can confirm I do get this with just import torch and a hello world print.

Seems like pytorch takes a long long time to compile too. Is there any way that I can make it only compile the parts I need it to instead of torch as a whole?

[kayhayen]

Yes, checkout anti-bloat plugin, it's extremely powerful. Would also be kind if you shared what you get away with in terms of enforcing imports to not be followed. It will give you warnings and errors for modules included that you don't like with pointers to the source, and anti-bloat.yml would love to have PRs.

[kayhayen]

What I find, is that this is probably because as a compile method, when provided as __init_subclass__ it is not annotated as a classmethod by Python core. Going to update the Torch plugin to do that. It's much like the __new__ and others, where Nuitka has to add these during their creation, as the type base class checks for pure Python function. Adding @classmethod in the source ought to do the trick. I might even include it in a hotfix, although I suspect their will be more issues with torch.

[joeyballentine]

Well if I run into any more issues after this is fixed I'll let you know.

And as for the anti-bloat plugin, it seems like it mainly only gets rid of testing related imports. Is there configuration I can do besides --no-follow-import in order to ignore imports? Certain ones like PyQt5 and scipy seem to get imported no matter what I configure.

[kayhayen]

@JoeyBallentine it has a more powerful version of the ignoring, which tells you where that happens. You provide custom rules, and can make it an error when it includes something you don't want to to include. Then you can tell why that happens. The anti-bloat can then through its configuration avoid it. But the nofollow ought to work actually. If you enable the tracing for it, --show-module I think it also says why something is decided.

[joeyballentine]

Any update on this? I've managed to remove a lot of bloat from the compilation, but sadly I still get this issue

[kayhayen]

It would be helpful if you shared your findings, so I can use that myself when I work on this. Right now I am not yet settled on how to make the modification in a sane way.

[joeyballentine]

I was able to get away with this for sure, though there is probably more than can be excluded tbh. But this at least sped up my compilation a lot:

python -m nuitka --mingw64 --standalone --lto no run.py \
--plugin-enable=torch --plugin-enable=pylint-warnings --plugin-enable=numpy --enable-plugin=anti-bloat \
--noinclude-pytest-mode=nofollow --noinclude-setuptools-mode=nofollow \
--noinclude-matplotlib --noinclude-scipy \
--nofollow-import-to=PyQt5 \
--nofollow-import-to=matplotlib \
--nofollow-import-to=scipy \
--nofollow-import-to=tkinter \
--nofollow-import-to=torchvision \
--nofollow-import-to=torchaudio \
--nofollow-import-to=IPython \
--nofollow-import-to=jedi \

[kayhayen]

ok, that's really a not to exclude scipy, I think. Generally making a minimal virtualenv is the recommended way, but I don't know if that fits in. I fear, tkinter is not obeyed yet. And IPython, should be really on that anti-bloat list, and jedi is probably caused by it, but some things pull it in too. But sure, anything that gives faster turnaround, will make it more workable for me to resolve the hidden imports from extension modules of torch itself, and there are still some.

[joeyballentine]

The biggest improvement imo was ignoring torch vision and torch audio, which it includes by default even if you don't use them. Since they aren't part of the main torch package, they can be ignored without causing crashes (as far as I can tell). This was the biggest source of improvement.

[kayhayen]

So, anti-bloat is great as of 1.7 and also with new torch2, so this can be considered resolved.