Include binary wheels for ARM Windows

[JukkaL]

Feature

We only ship x64 Windows binary wheels (compiled using mypyc). Maybe we should also ship ARM Windows wheels?

To make this work, we'd probably need help from somebody who has access to an ARM Windows system, at least for testing and perhaps benchmarking.

Pitch

Binary wheels compiled with mypyc could improve performance on ARM Windows systems by 3x to 4x or so. CPython already has an experimental installer for ARM Windows. The latest Microsoft Surface laptops have ARM CPUs, and it would be nice if performance was comparable to x64 on them.

Open questions

• Is Windows ARM popular enough among developers to make this worth it?
• Does GitHub Actions make it easy to build ARM binaries for Windows?
• Are all our dependencies supported on ARM Windows?

[ssbarnea]

Basedo on https://github.blog/news-insights/product-news/arm64-on-github-actions-powering-faster-more-efficient-build-systems/ arm64 windows runners are generally available, so I do not see any reason for not producing them. MacOS users are high likely to use an arm64 Windows installed on their machine instead of x86 one.

[JukkaL]

To move this forward, somebody with access to ARM Windows would have to try compiling mypy and mypyc and ensuring that tests pass and performance is reasonable. We can then look into enabling wheel builds.

[cdce8p]

Is Windows ARM popular enough among developers to make this worth it?

The current PyPI download stats for mypy 1.14.* within the last 28 days.

pypinfo -pc --days 28 --all --limit 100 \
  --where 'details.cpu != "None"' 'mypy==1.14.*' system cpu libc

system_name	cpu	libc_name	percent	download_count	Wheels
Linux	x86_64	glibc	85.90%	8,843,136	x
Windows	AMD64	None	5.13%	528,260	x
Linux	aarch64	glibc	3.26%	335,467	x (added with 1.15)
Darwin	arm64	None	3.04%	312,831	x
Linux	x86_64	None	1.31%	135,131	x
Darwin	x86_64	None	0.84%	86,298	x
Linux	x86_64	musl	0.37%	38,419	x
Linux	aarch64	None	0.06%	6,564
Linux	i686	glibc	0.02%	1,711
Linux	ppc64le	glibc	0.02%	1,571
Linux	aarch64	musl	0.01%	1,070
Linux	armv7l	glibc	0.01%	931
Emscripten	wasm32	glibc	0.01%	778	x (only 3.10)
Linux	s390x	glibc	0.01%	554
Linux	i686	None	0.00%	367
CYGWIN_NT-10.0-20348	x86_64	None	0.00%	305
Linux	armv7l	None	0.00%	276
FreeBSD	amd64	None	0.00%	185
Linux	ppc64le	None	0.00%	172
Windows	ARM64	None	0.00%	165	<--
Windows	x86	None	0.00%	88
NetBSD	amd64	None	0.00%	70
OpenBSD	amd64	None	0.00%	67
Linux	riscv64	glibc	0.00%	66
Linux	armv8l	None	0.00%	37

Does GitHub Actions make it easy to build ARM binaries for Windows?

Windows ARM runners aren't available just yet. Probably soon though https://github.com/github/roadmap/issues/1098 https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Are all our dependencies supported on ARM Windows?

So far I haven't seen any ARM Windows wheels on PyPI. That will likely only begin to change after the Github runners are available.

[JukkaL]

Thanks for looking up the stats! Based on them, this doesn't look relevant yet. We can revisit this once Windows ARM accounts for a non-trivial fraction of downloads (say 0.1%).

[finnagin]

Since arm64 runners are now released for public repos I went ahead and took a crack at adding a windows arm64 test to tests.yml and then adding windows arm64 wheel builds to build.yml in mypyc/mypy_mypyc-wheels.

The arm64 tests all passed and didn't take any longer than the x64 windows tests. Also, all of the arm64 wheels successfully built and passed their tests run by cibuildwheel which again did not take any longer than the x64 ones.

I had to regenerate test-requirements.txt in my fork of the mypy repo and point the mypy_mypyc-wheels workflow at that fork when checking out mypy to get it to run because lxml has windows arm64 wheels for 6.0 but not 5.4. Since build.yml in mypy_mypyc-wheels uses the test-requirements.txt pinned to a specific commit in this repo would the process be to submit a PR here, and then after merging, submit one to mypy_mypyc-wheels pinning it to the commit from that merged PR? (assuming you were open to a PR supporting windows arm64 wheels now)

[cdce8p]

I had to regenerate test-requirements.txt in my fork of the mypy repo and point the mypy_mypyc-wheels workflow at that fork when checking out mypy to get it to run because lxml has windows arm64 wheels for 6.0 but not 5.4. Since build.yml in mypy_mypyc-wheels uses the test-requirements.txt pinned to a specific commit in this repo would the process be to submit a PR here, and then after merging, submit one to mypy_mypyc-wheels pinning it to the commit from that merged PR? (assuming you were open to a PR supporting windows arm64 wheels now)

The process sounds about right. I just opened #19539 to update the test dependencies. However, I'm still not fully convinced it's worth it to add Windows ARM yet. Just looked at the current stats

pypinfo -pc --days 28 --all --limit 100 \
  --where 'details.cpu != "None"' 'mypy==1.17.*' system cpu libc

system_name	cpu	libc_name	percent	download_count	Wheels
Linux	x86_64	glibc	84.76%	6,341,745	x
Linux	aarch64	glibc	5.05%	377,825	x
Windows	AMD64	None	4.57%	341,925	x
Darwin	arm64	None	3.76%	281,077	x
Linux	x86_64	None	0.75%	56,458	x
Darwin	x86_64	None	0.67%	50,500	x
Linux	x86_64	musl	0.31%	23,109	x
Linux	aarch64	None	0.05%	3,857
Linux	aarch64	musl	0.04%	2,924
Linux	ppc64le	glibc	0.01%	417
Windows	ARM64	None	0.01%	380	<--
Linux	i686	glibc	0.00%	311
Linux	armv7l	glibc	0.00%	281
FreeBSD	amd64	None	0.00%	215
Linux	s390x	glibc	0.00%	179
Emscripten	wasm32	None	0.00%	148	x (only 3.10)
Linux	armv7l	None	0.00%	86
...	...	...	0.00%	..
Total				7,481,853

Yes, the count has roughly doubled since February. It's still way below 500 installs though. IMO 0.1% might be a good threshold. Before that, it's probably just wasted CI time. Especially since the wheel build is run for every commit.

[finnagin]

It's still way below 500 installs though. IMO 0.1% might be a good threshold.

Yeah, that makes complete sense. Though from the packaging side of things would it be worth it to support arm64 windows wheels for other projects that have a mypy/mypyc dependency? That way those projects could be unblocked to produce their own arm64 native wheels.

[cdce8p]

Though from the packaging side of things would it be worth it to support arm64 windows wheels for other projects that have a mypy/mypyc dependency? That way those projects could be unblocked to produce their own arm64 native wheels.

You can still run mypyc with the pure Python wheel. It would just be a bit slower. It's even possible to compile mypy one the fly, haven't tested it but it should be something like this

MYPY_USE_MYPYC=1 pip install --no-binary mypy mypy

[cdce8p]

It might make sense to reevaluate this now. This morning I was a post on which explains why the usage is so low currently. https://discuss.python.org/t/python-on-windows-arm64/104524

Basically the installer will still default to amd64 with emulation instead of ARM64 as there aren't that many ARM specific wheels just yet. The Python windows team plans to change the default in time for the Python 3.15 release next year.

With that I think it would make sense to start publishing arm wheels now. Opened https://github.com/mypyc/mypy_mypyc-wheels/pull/106

Open question

@JukkaL What's the current status on the PyPI project size limit? The last limit increase was two years ago and if we start publishing Windows ARM wheels, 6 more per release, and possibly free-threading wheels at the end of the year, 14 more per release, we might hit the limit fairly soon.

-- I've opened https://github.com/mypyc/mypy_mypyc-wheels/pull/106 for the wheel builder.

[cdce8p]

Open question

What's the current status on the PyPI project size limit? The last limit increase was two years ago and if we start publishing Windows ARM wheels, 6 more per release, and possibly free-threading wheels at the end of the year, 14 more per release, we might hit the limit fairly soon.

@JukkaL Any update here?

[JukkaL]

Yeah, it may make sense to propose increasing the PyPI size limit soon. Does anybody know where we can see how close we are to the 20 GB limit?

[cdce8p]

Does anybody know where we can see how close we are to the 20 GB limit?

After login in to PyPI, navigate to Your Projects -> mypy -> Manage and then Settings. This like might also work https://pypi.org/manage/project/mypy/settings/

[JukkaL]

We're at 17.3 GB / 20 GB, so an increase will be needed soon.

[cdce8p]

We're at 17.3 GB / 20 GB, so an increase will be needed soon.

Went ahead and created https://github.com/pypi/support/issues/8398 for it.