windows installer generator broken

[petersilva]

On ubuntu 18.04, it fails without a python interpreter version I can invoke ./generate-win-installer 3.6.8 and get a package generated successfully. (note it also fails on 18.04 if I do not specify the python interpreter version.)

I cannot determine any combination of settings that generates a package on ubuntu 20.04. Variety of different failure modes to be indicated in future posts to this bug. just a place marker for now as a worry about sustainability of this method.

Using 22.04, now, for versions (spring 2023) 3.00.43, and (summer 2023) 3.00.44, the installer generation was working again, but for 3.00.45 (fall 2023) it broke again. It isn't reliable enough to depend on.

[petersilva]

example 1:

fractal% ./generate-win-installer.sh

Collecting appdirs
 Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
 Saved ./appdirs-1.4.4-py2.py3-none-any.whl
Successfully downloaded appdirs
ERROR: Could not find a version that satisfies the requirement netifaces (from versions: none)
ERROR: No matching distribution found for netifaces
Collecting pika
 Using cached pika-1.2.0-py2.py3-none-any.whl (154 kB)
 Saved ./pika-1.2.0-py2.py3-none-any.whl
Successfully downloaded pika

  .
  .
  .
Collecting wheel file: PyNaCl-1.4.0-cp38-cp38-win_amd64.whl (from: pynsist_pkgs/*.whl)
Traceback (most recent call last):
 File "/home/peter/.local/bin/pynsist", line 10, in <module>
   sys.exit(main())
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/__init__.py", line 520, in main
   ec = InstallerBuilder(**args).run(makensis=(not options.no_makensis))
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/__init__.py", line 473, in run
   self.prepare_packages()
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/__init__.py", line 347, in prepare_packages
   wg.get_all()
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/wheels.py", line 296, in get_all
   self.get_globs()
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/wheels.py", line 314, in get_globs
   self.validate_wheel(path)
 File "/home/peter/.local/lib/python3.8/site-packages/nsist/wheels.py", line 335, in validate_wheel
   raise ValueError('Wheel {} is not compatible with Python {}, {}'
ValueError: Wheel PyNaCl-1.4.0-cp38-cp38-win_amd64.whl is not compatible with Python 3.8.5, win_amd64

So there are two problems (so far): It cannot locate a wheel for one dependency (netifaces) and it declares a wheel that should satisfy another one Pynacl to be incompatible.

tried giving specific python versions as argument to script 3.8.0, 3.8.1 ... 3.8.8 same result. >8 gaven an unknown URL presumably those versions do not exist yet.

[takluyver]

netifaces appears to have not made a release for a couple of years, so it doesn't have a pre-compiled wheel for Python 3.8: https://github.com/al45tair/netifaces/issues/55

[takluyver]

You might also want to put set -e at the top of your bash script, so it stops if one of those pip download commands fails, rather than carrying on and trying to build with pieces missing.

[petersilva]

Is it possible that this is just the netifaces developer thinking cp36 is good for >= 3.6 just like the other packages? If there is no higher version, should it not just pull in the 3.6 one.. the fact that it doesn't contradicts the work you are doing for other packages (Cryptography)

[takluyver]

I'm afraid not. This is the difference between the cp36-abi3 tags which Cryptography uses and the cp36-cp36m style tags which netifaces (and most packages with compiled parts) use. cp36m means it's built specifically against Python 3.6 and only compatible with that, whereas abi3 indicates it uses the CPython stable ABI, so the same compiled modules should be compatible with various versions of Python.

This is also why netifaces has Github issues open asking for wheels for Python 3.8 and 3.9.

[takluyver]

If you want a quick solution, someone appears to have made a fork just to put a Python 3.8 wheel on PyPI: https://pypi.org/project/netifaces-w38/#files . But using that means trusting that the random person who built it hasn't added something malicious to the package.

It's probably unlikely that any given package is malicious, but this would be a very easy way to carry out a supply-chain attack: find a widely used package that needs compilation but has stopped making pre-built packages, upload newer packages that work but also contain a backdoor, then sit back and wait for people to use them. There's no need to hack anything and it's not easy to detect, at least until the backdoor is put to use, which could be once it's already on thousands of systems.

[petersilva]

Thanks for the suggestion. I have created a package using that. Trying it out (the netifaces stuff works, but have not been able to confirm that the cryptography part works... need an sftp upstream, which is not in my test environnment.) The pynsist package is here:

http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/Sarracenia_2.21.03post2.exe

You can play with it if you want.

[petersilva]

A lot of work was done to get a working windows executable v3.0.44, and we got a new version for that:

https://github.com/MetPX/sarracenia/releases/tag/v3.00.44

There is a self-extracing executable that was kind of working for that version,

but it has a problem with log rotation ( #785 ) so you need to stop it, then rotate the logs, then start it up again... (After working on #748, #753, and many other fixes in release 3.00.43.

I went to create a binary for 3.00.45... it fails again... some dependency issue.

[petersilva]

sample run of the procedure (procedure is here:

• https://metpx.github.io/sarracenia/Contribution/Development.html#building-a-windows-installer

sample run:

fractal% ./generate-win-installer.sh 3.10.11 
   .
   .
  .
    self.prepare_packages()
  File "/home/peter/.local/lib/python3.10/site-packages/nsist/__init__.py", line 362, in prepare_packages
    copy_modules(self.packages, build_pkg_dir,
  File "/home/peter/.local/lib/python3.10/site-packages/nsist/copymodules.py", line 165, in copy_modules
    mc.copy(modname, target, exclude)
  File "/home/peter/.local/lib/python3.10/site-packages/nsist/copymodules.py", line 115, in copy
    raise ImportError('Could not find %r' % modname)
ImportError: Could not find 'pathtools'
fractal% 

pathtools is not a known dependency of the package looking into the elided log... ok found the problem... it looks fixed again...