macOS: added README note about Python version

[isaac-mcfadyen]

Hey! 👋

Tested out building workerd today and ran into a small issue. The build fails on Python 3.9.7, and since miniforge has that as the default version (when no environments are activated) this seems like a common issue.

The solution is to use Python 3.10. Because I'm using miniforge I simply created a new Conda environment with Python 3.10 and activated it before build and that solved the issue.

This PR adds a note to the docs (under the macOS section) about this restriction/limitation.

[kentonv]

Hi @isaac-mcfadyen,

What was the error under 3.9? I suppose this probably applies to Linux as well?

[kentonv]

Hmm, actually, I have 3.9.2 installed on Debian and haven't seen an issue.

[isaac-mcfadyen]

Hmm, interesting. Might also be related to ARM64, since I'm on an M1 Mac (ARM)?

I'm also on macOS Ventura 14 (Developer Beta) but I don't think that was it... although I might be wrong 😄

Full stacktrace follows (went away when moving to Python 3.10):

ERROR: /private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/workerd-v8/BUILD.bazel:1:11: @workerd-v8//:v8 depends on @v8//:v8_icu in repository @v8 which failed to fetch. no such package '@v8_python_deps//': rules_python_external failed:  (ERROR: Exception:
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/commands/wheel.py", line 158, in run
    resolver.resolve(requirement_set)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/legacy_resolve.py", line 201, in resolve
    self._resolve_one(requirement_set, req)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/legacy_resolve.py", line 365, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/legacy_resolve.py", line 311, in _get_abstract_dist_for
    req.populate_link(self.finder, upgrade_allowed, self.require_hashes)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/req/req_install.py", line 225, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/index.py", line 878, in find_requirement
    best_candidate_result = self.find_best_candidate(
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/index.py", line 861, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/index.py", line 791, in find_all_candidates
    collected_links = self._link_collector.collect_links(project_name)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/collector.py", line 541, in collect_links
    for page in self._get_pages(url_locations):
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/collector.py", line 493, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/collector.py", line 339, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/collector.py", line 129, in _get_html_response
    resp = session.get(
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_internal/network/session.py", line 426, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/cachecontrol/adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/connection.py", line 368, in connect
    self.ssl_context = create_urllib3_context(
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/util/ssl_.py", line 286, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/pypi__pip/pip/_vendor/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
    self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1128, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/OpenSSL/SSL.py", line 359, in __init__
    self.callback = _ffi.callback(
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/rules_python/python/pip_install/extract_wheels/__main__.py", line 5, in <module>
    main()
  File "/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 87, in main
    subprocess.run(pip_args, check=True)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/Caskroom/miniforge/base/bin/python3', '-m', 'pip', 'wheel', '-r', '/private/var/tmp/_bazel_isaac/a327433c429979ff29f013e2923f6d65/external/v8/bazel/requirements.txt', '--require-hashes']' returned non-zero exit status 2.
)

[kentonv]

Ah, I see, this looks like it could indeed be specific to Mac M1.

[kentonv]

Just to check, @mrbbot @penalosa, what versions of Python do you have on your macs? Just want to verify that 3.10 seems to be required generally and this isn't a problem specific to @isaac-mcfadyen's setup...

[mhart]

Works for me with Python 3.9.10 (under /opt/homebrew/bin/python)

[mhart]

Er, sorry, sent that from the wrong computer. Try again: Works for me with Python 3.9.6 (under /usr/bin/python3)

[mhart]

It sounds like it might be a PyOpenSSL problem (possibly with something to do with libffi too) – specific to certain setups: https://github.com/pyca/pyopenssl/issues/873

[isaac-mcfadyen]

It sounds like it might be a PyOpenSSL problem

Interesting, sounds like that could be it! Right down to the "fix" of upgrading to 3.10 described in this comment and a couple others.

[mhart]

In both my cases I was using either system python, or regular homebrew python – could it be something to do with conda or one of the packages in your env?

[mrbbot]

what versions of Python do you have on your macs?

This setup was working for me. Presumably Homebrew 3.10 Python is being used by the build here.

$ python3 --version
Python 3.10.6

$ where python3
/opt/homebrew/bin/python3
/usr/bin/python3

$ /opt/homebrew/bin/python3 --version
Python 3.10.6

$ /usr/bin/python3 --version         
Python 3.8.9

[penalosa]

Mine is the same as @mrbbot —homebrew Python 3.10

[isaac-mcfadyen]

PyOpenSSL is installed in my (base) Conda env, could it be that? pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge