Failed to Build on Python 3.11.0

[shubhamshah02]

OS/Arch

$ python -c 'import platform;print(platform.uname())'

Python version

3.11.0

$ python --version

cChardet version

Latest

$ python -c 'import cchardet;print(cchardet.__version__)'

What is the problem?

Failed to build.

Expected behavior

Supposed to build.

Actual behavior

fatal error: longintrepr.h: No such file or directory

Steps to reproduce the behavior

Just update to Python 3.11 and attempt to build code results in error.

[ToasterUwU]

Same here.

      src/cchardet\_cchardet.cpp(196): fatal error C1083: Cannot open include file: 'longintrepr.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cchardet

[ToasterUwU]

nvm that issue mention, i made a little mistake and assumed that orjson depends on this, which it doesnt.

[lvillis]

Same here.

      src/cchardet\_cchardet.cpp(196): fatal error C1083: Cannot open include file: 'longintrepr.h': No such file or directory
      error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.32.31326\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cchardet

[seperman]

Same here

[SimplicityGuy]

There are 2 PRs #78 and #80 that will address this. @PyYoshi, can you merge and release a new build, please?

[woctezuma]

I encounter the same issue with the cchardet package.

I have noticed it as I wanted to install ASF_IPC, which depends on two other packages:

aiohttp
cchardet

For the aiohttp package, the issue was solved in September:

• https://github.com/aio-libs/aiohttp/issues/6600

[ToasterUwU]

@PyYoshi Doesnt seem to come back to take care of this. Tried to contact them on all platforms I could find them on, never got any response.

Its also well known that they don't maintain this, it has been like this for literal years.

So chances for an update are very slim. I recommend everyone who uses this, to find an alternative.

[woctezuma]

It is definitely a bit weird that the requirements mention chardet v3.0.4 when the current version is v5.0.0, which was updated for Python v3.11.

However, I think one would have to also change other requirements, notably the version of Cython.

[seperman]

Someone should fork this repo and start publishing it on Pypi as cchardet2.

Sep Dehpour

On Nov 19, 2022, at 6:52 AM, Wok @.***> wrote:

 It is definitely a bit weird that the requirements mention chardet v3 when the current version is v5, which was updated for Python v3.11.

However, I think one would have to also change other requirements, notably the version of Cython.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[ToasterUwU]

@seperman There is also a process that allows to transfer Pypi projects to a new user when the old owner is inactive.

Don't know exactly how it works, but I know it exists

[seperman]

That process needs it to go through some pypi committee voting to change ownership. I recently lost my 2 factor auth to pypi. In order to get back access to my account, Pypi took about eight months. It didn't even need to go through a committee, and it took that long.

Sep Dehpour

On Nov 19, 2022, at 11:11 AM, ToasterUwU @.***> wrote:

 @seperman There is also a process that allows to transfer Pypi projects to a new user when the old owner is inactive.

Don't know exactly how it works, but I know it exists

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[ToasterUwU]

Even if it takes long, it should still be done.

Throwing it away and making a new package is not a good idea in my opinion. Since it forces everyone who depends on it to figure out there is a new one and than switch.

[seperman]

Do you want to wait a couple of years until the PyPI committee votes for it? You can start the process now but in the meantime, have cchardet2 on pypi. Personally if I can not use it with Python 3.11 by the end of the month, I will simply stop using cchardet altogether.

Sep Dehpour

On Nov 20, 2022, at 1:00 AM, ToasterUwU @.***> wrote:

 Even if it takes long, it should still be done.

Throwing it away and making a new package is not a good idea in my opinion. Since it forces everyone who depends on it to figure out there is a new one and than switch.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[ToasterUwU]

@seperman Im neither attached enough to take this project under my wings, or enough of a C nerd to even be able to maintain this.

Besides that, Pypi will not take years to do this. Definetly not. I heard a few stories about people needing to use this option, and from what I heard, it was only a matter of months.

Multiple months is still quiet some time. But just slapping a 2 behind the name and than having the normal one still exist and look like the better option on first glance is not a better option than waiting. I agree there should be a new package for the meantime. But its crucial that the package with the most trustworthy name and most downloads and all that, is taken over as soon as its possible.

[wbarnha]

We use this project for https://github.com/faust-streaming/faust and we're trying to migrate everything to support Python 3.11. I am willing to take a try at upgrading things in this project, in my own fork, for Python 3.11.

[seperman]

@wbarnha Cool. Please let us know once you fork it.

[seperman]

@ToasterUwU The last time @PyYoshi RIP pushed any code to this code base seems to be 2 years ago.

[wbarnha]

I've made a fork at https://github.com/faust-streaming/cChardet and I'm going to merge the pending PRs into it.

[wbarnha]

@seperman wheels for cchardet on Python 3.11 are now available!

pip install faust-cchardet

Sadly, wheels for Windows are not available at the moment until I figure out what's going on with https://github.com/faust-streaming/cChardet/actions/runs/3568007197.

Edit: I made a mistake and forgot to build Python 3.11 wheels. Should be fixed momentarily.

[seperman]

Awesome! Thanks.

[MathieuDuponchelle]

@wbarnha thanks for taking this on, can you open issues on your fork? I'm encountering various problems, such as when installing via a setup.py install_requires:

Searching for faust-cchardet
Reading https://pypi.org/simple/faust-cchardet/
Downloading https://files.pythonhosted.org/packages/68/18/923a6257a9bbc40d1fbbf35740f823472ca73d7ff13939be9e5c7cbdfbf5/faust-cchardet-2.1.10.tar.gz#sha256=b5f50e2fa4256a3a237f2e527e212d85130117d58e4bd1a0e729bc7920634ecb
Best match: faust-cchardet 2.1.10
Processing faust-cchardet-2.1.10.tar.gz
Writing /tmp/easy_install-xphxe8vb/faust-cchardet-2.1.10/setup.cfg
Running faust-cchardet-2.1.10/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xphxe8vb/faust-cchardet-2.1.10/egg-dist-tmp-f_de48lz
warning: no files found matching '*.pxd' under directory 'src'
warning: no files found matching '*.pxi' under directory 'src'
warning: no files found matching '*.cpp' under directory 'src'
warning: no files found matching '*' under directory 'src/ext'
cc1plus: fatal error: src/ext/uchardet/src/CharDistribution.cpp: No such file or directory

And when installing manually:

$ python3 -c "import cchardet"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/meh/.local/lib/python3.10/site-packages/cchardet/__init__.py", line 1, in <module>
    from cchardet import _cchardet
ImportError: dynamic module does not define module export function (PyInit__cchardet)

OS is Linux (Fedora 36), python version 3.10.8

[wbarnha]

I created an issues tab in my fork, sorry it wasn't already there. I'll take a look when I get the opportunity.

[MathieuDuponchelle]

Thanks a bunch @wbarnha , and yeah I think by default github doesn't create an issues tab on new repos, no reason to be sorry :)

I'll file the two issues later today.

[stromnov]

cchardet for Python 3.11 requires recent version of Cython at build phase to eliminate longintrepr.h issue

[yves-renier]

I confirm that a pip install --upgrade Cython fix the issue. Thank you @stromnov

[ayratzay]

This update (from 2.0.4) helped me with the issue. python -m pip install charset-normalizer==2.1.0

[lahwaacz]

One year later and this is still a problem... :disappointed:

[Mohsen-Amiri]

cchardet for Python 3.11 requires recent version of Cython at build phase to eliminate longintrepr.h issue

I installed Cython but cannot install twint yet. Do you have any idea helps me?

[xloc]

python 3.11.6 with Cython 3.0.6 works for me. Please double check your environment

[vgavro]

@xloc For same versions, still same error. python3.11.6, Cython 3.0.6 on installing from pypi! UPD: BUT from git everything ok for some reason!

vgavro:~/w/extractnet(venv)(master+%=)> python3 -m venv ./venv
vgavro:~/w/extractnet(venv)(master+%=)> ./venv/bin/python --version
Python 3.11.6
vgavro:~/w/extractnet(venv)(master+%=)> cat /etc/*release
DISTRIB_ID="Arch"
DISTRIB_RELEASE="rolling"
DISTRIB_DESCRIPTION="Arch Linux"
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo
vgavro:~/w/extractnet(venv)(master+%=)> ./venv/bin/pip install cython==3.0.6
Collecting cython==3.0.6
  Obtaining dependency information for cython==3.0.6 from https://files.pythonhosted.org/packages/4b/5b/d51fa7c329e278ab53d24ee0837a77993244d48f5c6d7c159844ca475f5b/Cython-3.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading Cython-3.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
Downloading Cython-3.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 6.3 MB/s eta 0:00:00
Installing collected packages: cython
Successfully installed cython-3.0.6
vgavro:~/w/extractnet(venv)(master+%=)> ./venv/bin/pip install cchardet
Collecting cchardet
  Using cached cchardet-2.1.7.tar.gz (653 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cchardet
  Building wheel for cchardet (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for cchardet (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/cchardet
      copying src/cchardet/version.py -> build/lib.linux-x86_64-cpython-311/cchardet
      copying src/cchardet/__init__.py -> build/lib.linux-x86_64-cpython-311/cchardet
      running build_ext
      building 'cchardet._cchardet' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/cchardet
      creating build/temp.linux-x86_64-cpython-311/src/ext
      creating build/temp.linux-x86_64-cpython-311/src/ext/uchardet
      creating build/temp.linux-x86_64-cpython-311/src/ext/uchardet/src
      creating build/temp.linux-x86_64-cpython-311/src/ext/uchardet/src/LangModels
      gcc -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -Isrc/ext/uchardet/src -I/home/vgavro/work/extractnet/venv/include -I/usr/include/python3.11 -c src/cchardet/_cchardet.cpp -o build/temp.linux-x86_64-cpython-311/src/cchardet/_cchardet.o
      src/cchardet/_cchardet.cpp:196:12: fatal error: longintrepr.h: No such file or directory
        196 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cchardet
Failed to build cchardet
ERROR: Could not build wheels for cchardet, which is required to install pyproject.toml-based projects

 ./venv/bin/pip install git+https://github.com/PyYoshi/cChardet
Collecting git+https://github.com/PyYoshi/cChardet
  Cloning https://github.com/PyYoshi/cChardet to /tmp/pip-req-build-y6txjczj
  Running command git clone --filter=blob:none --quiet https://github.com/PyYoshi/cChardet /tmp/pip-req-build-y6txjczj
  Resolved https://github.com/PyYoshi/cChardet to commit bd1cc39950f60fea2773cbfc587c9395e85fe6f3
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cchardet
  Building wheel for cchardet (pyproject.toml) ... done
  Created wheel for cchardet: filename=cchardet-2.1.7-cp311-cp311-linux_x86_64.whl size=196658 sha256=4944e02fb0d6d11caba507519a658aec3e56b1fe42a816bdd0699fd5d415b5d9
  Stored in directory: /tmp/pip-ephem-wheel-cache-bws5po5i/wheels/7c/70/41/0a677f278fd9d41235ddbde99ac020e0488fc3b2db384d26a5
Successfully built cchardet
Installing collected packages: cchardet
Successfully installed cchardet-2.1.7