pip install notcurses fails on Windows 10 Pro

[pjfarley3]

Please include the following data:

• export | egrep 'LANG|LC_CTYPE|TERM'
• notcurses version (available from notcurses-demo i)
• terminal name + version

C:\Users\Myuser\myproject>set | egrep "LANG|LC_CTYPE|TERM" LANG=EN LANGUAGE=EN notcurses install from PyPi (3.0.8) Microsoft Windows Terminal Version: 1.13.11431.0

Running on this Windows version:

OS Name: Microsoft Windows 10 Pro OS Version: 10.0.19044 N/A Build 19044

Tried installing notcurses in a pipenv virtual shell today, got the attached (sanitized for private info) error output, the start of which is as follows:

(myproject-abcdefgh) C:\Users\Myuser\myproject>pip install notcurses
Collecting notcurses
  Downloading notcurses-3.0.8.tar.gz (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.6/67.6 KB 1.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [31 lines of output]
      C:\Users\Myuser\.virtualenvs\myproject-abcdefgh\lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
<python traceback lines here>
        File "src/notcurses/build_notcurses.py", line 15, in <module>
          lines = fp.read().splitlines()
        File "C:\Python310\lib\encodings\cp1252.py", line 23, in decode
          return codecs.charmap_decode(input,self.errors,decoding_table)[0]
      UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2472: character maps to <undefined>
      warning: pypandoc module not found, won't generate man pages
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Do I need to install pypandoc before installing notcurses? Shouldn't that be in the requirements.txt file as a requisite package?

Or is there a different problem here?

The tail end of the error log also contains my python virtual environment info, repeated here for reference:

(myproject-abcdefgh) C:\Users\Myuser\myproject>pip list
Package           Version
----------------- -------
cffi              1.15.1
freetype-py       2.3.0
numpy             1.23.0
pip               22.0.4
pycparser         2.21
setuptools        60.10.0
tcod              13.6.2
typing_extensions 4.3.0
wheel             0.37.1

(myproject-abcdefgh) C:\Users\Myuser\myproject>python -V
Python 3.10.4

Peter

Error log: notcurses-install-error.log

[HexDecimal]

This build script opens include/notcurses/notcurses.h which seems to have UTF-8 encoded characters, but encoding="utf-8" is missing from the call to open.

Setting the environment variable PYTHONUTF8=1, or otherwise forcing Python's UTF-8 mode might help as a temporary workaround:

python -X utf8 -m pip install notcurses

[pjfarley3]

Thanks for the prompt reply. Using both options you gave (both the environment variable and the X -utf8 option to pip) it got a bit further along, but trying to build _notcurses.c (twice for some reason I do not understand) it could not find notcurses/direct.h. Log attached.

Error log: notcurses-install-error#2.log

[HexDecimal]

I think the source on PyPI assumes that a dev build of the Notcurses library exists, which is why it fails to include headers in the current log. I tried to install the newer Python implementation and had the same problem. The setup scripts try to link to an existing library instead of building it.

In that case neither of these Python implementations are portable, and it will be especially difficult to install a wheel on Windows. The current setup is ideal for Linux package managers but not for PIp or PyPI.

twice for some reason I do not understand

Compiling multiple sources at once is normal. Compile jobs get split across multiple processes so that they can use more CPU cores, and that means they can fail at the same time before the remaining jobs are canceled because of the errors.

[pjfarley3]

I think the source on PyPI assumes that a dev build of the Notcurses library exists, which is why it fails to include headers in the current log. I tried to install the newer Python implementation and had the same problem. The setup scripts try to link to an existing library instead of building it.

In that case neither of these Python implementations are portable, and it will be especially difficult to install a wheel on Windows. The current setup is ideal for Linux package managers but not for PIp or PyPI.

What exactly does that mean please? Would a python environment with an earlier version of python itself (say 3.8 or 3.9) be more likely to succeed? If there are libraries (i.e., requirements) necessary to a successful link, why aren't they in the requirements.txt file so that pip (or one of the setup processes) can at least complain that a user is missing that requirement?

I can easily reset my pipenv virtual environment to either python 3.8.10 or python 3.9.12 (I have both of those versions still installed, though I normally use the 3.10 version). Would that solve any of the problems? If so, a mention of the tested python build version on the PyPi page would be very helpful to interested users.

Are there other PyPi packages or libraries that need installation as prerequisites to installing notcurses? Any that I could search for and install myself?

As I was eagerly looking forward to trying out notcurses for my python project I would hope and pray that these issues could be resolved in a reasonable time frame (however that can be measured in volunteer time). I am not very good at git or github development procedures (having never done any of it), much less PyPi build processes, so I don't think my volunteer time would aid the project much.

twice for some reason I do not understand

Compiling multiple sources at once is normal. Compile jobs get split across multiple processes so that they can use more CPU cores, and that means they can fail at the same time before the remaining jobs are canceled because of the errors.

Of course, I had forgotten about the parallel compilation capabilities. I don't see many install processes that actually use it, so it surprised me. Thanks for the explanation.

[Edit] I tried setting the pipenv virtual environment to both 3.8 and to 3.9, and both generated the same "header not found" error as the 3.10 setup.

Please let me know if this issue is fundamental and will always fail on Windows, or if there is any light at the end of the tunnel that will eventually allow a Windows version to be installed and used.

[HexDecimal]

Would a python environment with an earlier version of python itself (say 3.8 or 3.9) be more likely to succeed? If there are libraries (i.e., requirements) necessary to a successful link, why aren't they in the requirements.txt file so that pip (or one of the setup processes) can at least complain that a user is missing that requirement?

The issue involves the C extension itself. C libraries are not handled directly by PyPI or Pip, so there isn't an easy Python solution. A different version of Python will not fix this, every version of Python is affected, since a broken source distribution was uploaded to PyPI. Broken because the sources do not include the files needed to build Notcurses itself, making the source distribution non-portable.

So if you want to continue you might have to clone the repository and modify the Python sources.

Please let me know if this issue is fundamental and will always fail on Windows, or if there is any light at the end of the tunnel that will eventually allow a Windows version to be installed and used.

The Python package is just immature. Many other packages had to deal with this exact issue. It's usually solved by statically linking the libraries involved.

If you don't have experience with Python's C extension build system and compiling/linking C then you will not easily solve this. A long term solution requires changes to the Python setup scripts.

Theoretically the minimal effort solution is to clone the repo, build the Notcurses library with CMake, then pass the include and library directories to Python as environment variables when running the setup script (or maybe insert them directly into the setup script.) This is close to how the package is being built on Linux/MacOS and fulfills the assumptions made by the current build script.

[pjfarley3]

Sorry for taking some time to reply; RL issues.

I do think I understand the python build issue, there is a similar case with the "windows-curses" PyPi project, which uses a custom offline build of the PDCurses library which is (I think) statically bound into a *.pyd file in the distribution on PyPi. When new Python releases come out (like the upcoming 3.11 release), you have to wait a bit before the "windows-curses" team (who actually work on a much larger project that uses it) "catch up" to the new python binary with a compatible binary library release.

I tried a pip3 install of notcurses in WLS1 Ubuntu 20.04 (python 3.8.10) available in my Windows system and got the same error at the same point there as on native Windows, so I guess there's just no option for me to use a python version until that gets resolved.

Thanks for your time and for your courteous, informative responses.