Adding Windows compatibility for hyperscan

[steven-t-murphy]

Discussed in https://github.com/darvid/python-hyperscan/discussions/155

^{Originally posted by steven-t-murphy September 9, 2024}

Our team has been working on a project that required Hyperscan to be compatible with Windows. We modified the setup.py and created a script to enable successful builds on Windows, and we are now using Hyperscan on both Windows and Linux platforms.

We would like to contribute these changes back to the community.

Exit Criteria:

• python-hyperscan fork are both public on GitHub
• GitHub regression testing operational
• pypi updated with windows compatibility

[JaneX8]

For this reason: https://github.com/SAP/credential-digger/issues/52#issuecomment-712261327

I would really like to see Windows support too. to be able to use https://marketplace.visualstudio.com/items?itemName=SAPOSS.vs-code-extension-for-project-credential-digger the plugin in VSCode.

[LucianoBAF]

@darvid

[JaneX8]

Does anyone happen to have any more information on the progress of this issue, or alternative solutions?

[darvid]

For credential digger specifically, it appears that extension supports changing the binary location as well as using a Docker container, so I would suspect it could be possible to use WSL or docker?

[JaneX8]

Ah yes. I could make that work. Thank you for that idea.

[steven-t-murphy]

PR opened to add windows compatibility https://github.com/darvid/python-hyperscan/pull/168

[JaneX8]

PR opened to add windows compatibility https://github.com/darvid/python-hyperscan/pull/168

Thank you Steven! Great job.

[steven-t-murphy]

To build the wheels I am currently bypassing pdm_build https://github.com/darvid/python-hyperscan/blob/30c13f55bb3a973a8b66e372dd6848c4a7f0b991/pdm_build.py#L43 which specifies "win32 not currently supported". The way I am bypassing is simply running.

python -m pip install build
python -m build --wheel

I think as we look at the CI toolset this may be an aspect to look at for enabling the build of Windows artifacts.

[steven-t-murphy]

@darvid curious if you have a timeline in mind for releasing 0.79, and if there is anything you need from me to support adding the windows wheels to pypi.

[darvid]

@steven-t-murphy apologies for the delays - spent a bit more time than anticipated due to factors mentioned below, but the long and short of it is I'd appreciate if you get a chance to checkout and build the wheels on Windows whenever you get a chance if you don't mind, with cibuildwheel ideally, e.g.:

pipx install cibuildwheel
cibuildwheel --only=cp313-win_amd64

TLDR of the changes, thanks to your work on the windows build that helped me figure out the following:

• updated the build process to use more modern standards for native Python extensions, e.g. via scikit-build-core (+ CMake)
• what took some time was figuring out how to build on win32 statically linked to MSVC and hyperscan. also, vectorscan explicitly dropped support for windows, so the current solution is use the last open source version of Hyperscan on win32, and vectorscan on other platforms (customizable with build time flags). an option for optimizing the build process would be to vendor HS/VS + build time deps for chimera/static support like PCRE, but for now I'm just pulling in dependencies with CMake.
• had to do a bunch of platform specific configuration for CMake, currently some stuff is hardcoded (e.g. platform version for visual studio is 17 2022 which is the version available on MS Server Github action runners) while I test all platforms; windows, linux (gcc, glibc), and linux (musl) - trying macOS x86 and apple silicon next.

if everything works for you, the next step would be to configure the CI which in theory should be less complex now as we wouldn't need platform specific build scripts moving forward.

[darvid]

ok there should be some windows wheels here:

https://github.com/darvid/python-hyperscan/actions/runs/13213324889

[darvid]

unless there's any glaring issues, i'll cut a release soon

[LucianoBAF]

Thanks for making the Windows compatibility!! Just to let you know that the Pypi package is still at 0.7.8. Would love to get version 0.7.9 in there to start using it from pip :). Thanks

[lazy-chesapeake]

Would love to get version 0.7.9 in there to start using it from pip

It looks like your deployment workflow failed, @darvid. The binaries built successfully, but it failed to publish to PyPi

I suspect you simply need to change the package name back to hyperscan, as it was before your workflows revamp in 0.7.8

[darvid]

Would love to get version 0.7.9 in there to start using it from pip

It looks like your deployment workflow failed, @darvid. The binaries built successfully, but it failed to publish to PyPi

I suspect you simply need to change the package name back to hyperscan, as it was before your workflows revamp in 0.7.8

[internally screaming]

apologies for the delay, win32 wheels should now be up, iterated on a few patch versions while debugging CI so we're going from 0.7.9 -> 0.7.12 but i don't want to mutate history so hopefully no more redundant releases required moving forward.

https://github.com/darvid/python-hyperscan/actions/runs/15048801362/job/42299607077