Run tests on Python 3.11 as well

[vlaci]

[vlaci]

Unfortunately LIEF doesn't get built on Python 3.11. It should work, once a new release is published from it, as they have bumped the required pybind version and also added Python 3.11 to the CI.

[qkaiser]

LIEF has a wheel for Python 3.11 since yesterday :) https://github.com/lief-project/LIEF/releases/tag/0.12.3

[qkaiser]

So hyperscan for Python does not support Python 3.11 and I don't think it will be ready soon given that the maintainer is looking for others to take over (see https://github.com/darvid/python-hyperscan/issues/44).

The only thing breaking with Python 3.11 is the absence of libch, which corresponds to libchimera in hyperscan.

I guess we could fork the hyperscan repo to work on 3.11 support and maybe vectorscan support (for ARM) when we have enough bandwidth.

[vlaci]

So hyperscan for Python does not support Python 3.11 and I don't think it will be ready soon given that the maintainer is looking for others to take over (see darvid/python-hyperscan#44).

The only thing breaking with Python 3.11 is the absence of libch, which corresponds to libchimera in hyperscan.

I guess we could fork the hyperscan repo to work on 3.11 support and maybe vectorscan support (for ARM) when we have enough bandwidth.

Chimera is a mess to compile, statically links to PCRE which has to be compiled using their own cmake wrapper without sufficient documentation on how to do it. Tried to add it to Nix as well, but ran into https://github.com/intel/hyperscan/issues/359 at the end so I gave up for the time being. It definitely should be optional to use Chimera in the python wrapper because the above mentioned difficulties.

Thought about implementing our own thin wrapper which could be generalized over hyperscan and vectorscan. We don't really use much of the API surface afterall.

[qkaiser]

@vlaci works if we pin hyperscan to 0.2.0. Nix has a problem with poetry now 🐍

[vlaci]

@vlaci works if we pin hyperscan to 0.2.0. Nix has a problem with poetry now snake

Here you are :)

[AkechiShiro]

Hey I'm packaging unblob for the AUR and I can't seem to get python-hyperscan packaged, trying to get it packaged seems to be quite hard, v0.2.0 is according to this issue https://github.com/darvid/python-hyperscan/issues/55 not compatible with Python 3.10 and currently on Arch we're by default on Python 3.11.

Here is the PKGBUILD script, I'm using to build python-hyperscan :

_pyname=hyperscan
pkgname=python-${_pyname}-git
pkgver=v0.4.0.r3.g056538c
pkgrel=1
pkgdesc='A CPython extension for the Hyperscan regular expression matching library.'
arch=('any')
url="https://github.com/darvid/python-$_pyname"
license=(GPL)
depends=(python hyperscan)
conflicts=()
provides=(python-hyperscan)
makedepends=(git python-setuptools python-pip python-build python-installer python-wheel poetry python-platformdirs python-poetry-plugin-export) 
checkdepends=()
source=("git+https://github.com/darvid/python-$_pyname")
sha256sums=(SKIP)

pkgver() {
  cd "python-$_pyname"
  git describe --long --abbrev=7 | sed 's/\([^-]*-g\)/r\1/;s/-/./g'
}

prepare() {
    git -C "${srcdir}/python-${_pyname}" clean -dfx
}

build() {
    cd "python-$_pyname"
    PYTHON_HYPERSCAN_STATIC=false python -m build --wheel --no-isolation
}

package() {
    cd "python-$_pyname"
    python -m installer --destdir="$pkgdir" dist/*.whl
}

I'm running into the same issue as here, mainly libch is missing, I take it that I need to try to lower python-hyperscan's version to 0.2.0 first, and then try to build and see what happens ?

See output of build :

==> Retrieving sources...
==> WARNING: Skipping all source file integrity checks.
==> Extracting sources...
  -> Creating working copy of python-hyperscan git repo...
Cloning into 'python-hyperscan'...
done.
==> Starting prepare()...
==> Starting pkgver()...
==> Starting build()...
* Getting build dependencies for wheel...
* Building wheel...
Package libch was not found in the pkg-config search path.
Perhaps you should add the directory containing `libch.pc'
Traceback (most recent call last):
Package 'libch', required by 'virtual:world', not found
  File "/build/python-hyperscan-git/src/python-hyperscan/setup.py", line 29, in <module>
    build(setup_kwargs)
  File "/build/python-hyperscan-git/src/python-hyperscan/build_hyperscan_extension.py", line 67, in build
    pkg_config_options = pkgconfig(["libhs", "libch"], static=static)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/python-hyperscan-git/src/python-hyperscan/build_hyperscan_extension.py", line 33, in pkgconfig
    subprocess.check_output(
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pkg-config', '', '--libs-only-l', 'libhs', 'libch']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/usr/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 56, in build_wheel
    return WheelBuilder.make_in(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 85, in make_in
    wb.build(target_dir=directory)
  File "/usr/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
    self._build(zip_file)
  File "/usr/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 179, in _build
    self._run_build_command(setup)
  File "/usr/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 217, in _run_build_command
    subprocess.check_call(
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/build/python-hyperscan-git/src/python-hyperscan/setup.py', 'build', '-b', '/build/python-hyperscan-git/src/python-hyperscan/build']' returned non-zero exit status 1.

ERROR Backend subprocess exited when trying to invoke build_wheel
==> ERROR: A failure occurred in build().
    Aborting...
==> ERROR: Build failed, check /home/username/project/opensource/aur/unblob-git/chroot/zwei/build

[vlaci]

You shouldn't need python-hyperscan, we are using another library, pyperscan now.

[AkechiShiro]

I'll remove this dependency all together then and package pyperscan, thanks