pyopenssl
pyopenssl copied to clipboard
pyca
MemoryError: Cannot allocate write+execute memory for ffi.callback()
For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Found originally here: https://github.com/scrapy/scrapy/issues/4117 Seems like pyOpenSSL issue, that's why creating it here.
Any advice or workaround is appreciated.
Traceback:
2019-10-31 20:24:51 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET https://xxx.yyy/robots.txt>: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Traceback (most recent call last):
.... skipped as non-relevant ...
File "/home/scrapy/env/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1709, in _makeContext
ctx.set_verify(verifyFlags, _verifyCallback)
File "/home/scrapy/env/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1103, in set_verify
self._verify_helper = _VerifyHelper(callback)
OS:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
FWIW, I get the same message, on Ubuntu 18.04.3. It actually results in the same stack trace even when I go to execute other things (like install unrelated packages via pip). Uninstalling pyOpenSSL (after the scrapy install) allowed other installations to proceed, showing that it was pyOpenSSL related, but of course scrapy and other dependants can't fully function without it.
Andrii's first link above suggests some workarounds, but not for Ubuntu.
Root of the issue might be a grsecurity override, in (say) a shared webhosting environment. Use of ffi.callback() is apparently frowned-upon now as insecure, and that link above has suggested workarounds.
The end of my stack trace, just to note slight variations in line numbers from the above:
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 286, in create_urllib3_context
context.verify_mode = cert_reqs
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1119, in set_verify
self._verify_helper = _VerifyHelper(callback)
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 338, in __init__
"int (*)(int, X509_STORE_CTX *)", wrapper)
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL's callback system.
Yeah, seems like there's a ready way to fix the code by changing to new-style "Extern Python" callbacks in CFFI. Here's another project that ran into the same issue and fixed it that way, with a real fast turnaround:
Issue: https://github.com/FSX/misaka/issues/72 Commit: https://github.com/FSX/misaka/commit/321357b192235438aaa94061626a1fd7c3b3c67b#diff-cd1259298a2efe394f2d1bbea8b0d39e
I'm not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI.
I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure's CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux).
Reading the answers above, does it mean every library using openssl has to made that kind of changes ?
When @tiran mentions "use another TLS library", is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ?
tl;dr it's a problem in pyOpenSSL. OpenSSL is fine.
I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python's standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL.
It's an implementation artifact of pyOpenSSL's glue code that wraps OpenSSL's C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks.
There are ways to work around the problem, but it's complicated and lots of work. Nobody has contributed a solution yet.
This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs.
Is there anyone who could take a look at this?
It would take a serious effort and redesign to address the issue. Somebody would have to replace all existing callbacks in PyCA cryptography and pyOpenSSL with a different design. It's a non-trivial effort, because the implementation must be thread-safe and multi subinterpreter-safe. The effort could easily take two weeks or more even for an expert.
At work we had similar issues with SELinux enforcement of read-only executable mmap pages. We followed a different approach and replaced pyOpenSSL with Python's ssl module.
@tiran Thanks for the comment! I'll suggest dbt/snowflake-connector do the same thing you folks did.
Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞
Same problem on M1 MacBook Pro. It's going to be year 2022 and Apple Silicon is still not supported.
That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful.
And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1.
FWIW, I've been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I've come back to it every few months to see if it was still crashing (and it has been).
After seeing @alex's comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.
- M1 Macbook Pro
- MacOS 12.0.1
- Python 3.10.0
- Scrapy==2.5.1
My test case has always been:
$ scrapy shell 'http://httpforever.com/' # <- succeeds
$ scrapy shell 'https://n8henrie.com' # <- crashes with `Cannot allocate write+execute memory for ffi.callback()`
For me, both seem to now work without error on http or https sites, so perhaps some of the others following this thread might recheck to see if it is still an issue for them or not.
I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips.
It works on both Python 3.10.0 and 3.9.9 (the scrapy shell). I need to install PyTorch, but PyTorch isn't working with Python 3.10 right now... And when I was tring to build pytorch from source with M1, pytorch dragged me back to install conda before compiling ,which only supports upto python 3.9.7 right now. It's like a dead loop...
Resolved under newest python 3.10.2 , using conda virturl env conda create -n py310 python=3.10 my laptop is Mac m1 .
@shaders:
Same thing happening on Python 3.8.9
Can you give a specific way to reproduce the problem? e.g. a short, standalone python script which reproduces it for you?
@candlerb I've updated requests module from 2.21 to 2.26 and the problem was gone (probably because requests module no longer uses pyssl: https://docs.python-requests.org/en/latest/community/updates/
2.26.0 (2021-07-13) The requests[security] extra has been converted to a no-op install. PyOpenSSL is no longer the recommended secure option for Requests. (#5867)
-requests==2.21.0
+requests==2.26.0
Figured it out following @anribras 's advice. M1 mac, had the same issue here. As @anribras says, I upgraded Python in my conda env to 3.10.2 and now this issue is resolved.
I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips.
I also meet that question! @Alexander Liu I think you can use mini-conda to install torch where python version is 3.8, and use python which version is 3.10 to install scrapy.
I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips.
I also meet that question! @Alexander Liu I think you can use mini-conda to install torch where python version is 3.8, and use python which version is 3.10 to install scrapy.
thanks！！！
We see this still in the test suites of several packages in Nix on darwin-aarch64 and the thing all affected packages has in common is async.
And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1.
I note your test suite doesn't have any asyncio examples.
(n.b. we see it on both python 3.9.12 and 3.10.4)
If there's a reliable reduced test case that reproduces this that's the next step in determining how to fix/work around the issue.
@reaperhulk , I am getting the above error when making a connection on snowflake. Sharing in case it is helpful for reproducibility:
# pip install snowflake-connector-python
import snowflake.connector
conn = snowflake.connector.connect(...)
Here's the full traceback
Unfortunately the issue here is that these errors don't reproduce with arbitrary Python on M1. I just tested that on my M1 Max running 3.10 from pyenv and had no error.
I didn't have this error before installing these packages, and I do now:
7093 pip3 install numpy
7094 pip3 install dataclasses-json
7095 pip3 install scipy
7108 pip3 install solidpython
I also installed conda:
7100 curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"\nbash Mambaforge-$(uname)-$(uname -m).sh
7101 -
7102 conda install -c conda-forge -c cadquery cadquery=2
Uninstalling the packages above did not resolve the issue, but removing ~/mambaforge resolved the issue.
ETA: I am on a M1, and the conda install failed.
I have a reproducible example, but I don't fully understand why it's happening.
First of all, I do NOT see the ffi.callback() error at all with pyenv's python versions, and I also do NOT see the error while using a virtual environment. I only see the error when using a formula's internal homebrew version of python that uses the homebrew libffi library.
Code sample - get_context_factory definition mostly taken from deluge's crypto_utils, and the other functions are cherry-picked from some twisted wrapper code (in particular the internet._sslverify._makeContext function):
#! /usr/bin/env python3
from OpenSSL import SSL
class OpenSSLCertificateOptions:
_contextFactory = SSL.Context
_context = None
def __init__(self):
self._options = (
SSL.OP_NO_SSLv2 | SSL.OP_NO_COMPRESSION | SSL.OP_CIPHER_SERVER_PREFERENCE
)
self._mode = SSL.MODE_RELEASE_BUFFERS
self.method = SSL.SSLv23_METHOD
def getContext(self):
if self._context is None:
self._context = self._makeContext()
return self._context
def _makeContext(self):
ctx = self._contextFactory(self.method)
ctx.set_options(self._options)
ctx.set_mode(self._mode)
verifyFlags = SSL.VERIFY_NONE
# It'd be nice if pyOpenSSL let us pass None here for this behavior (as
# the underlying OpenSSL API call allows NULL to be passed). It
# doesn't, so we'll supply a function which does the same thing.
def _verifyCallback(conn, cert, errno, depth, preverify_ok):
return preverify_ok
ctx.set_verify(verifyFlags, _verifyCallback)
def get_context_factory():
cert_options = OpenSSLCertificateOptions()
# this call is borked - ffi.callback
# import pdb; pdb.set_trace()
ctx = cert_options.getContext()
return cert_options
if __name__ == "__main__":
get_context_factory()
print("sample worked!")
Now, to actually see the error you have to jump through some hoops. I saw this error with the version of python that gets installed via my deluge-meta Homebrew formula (deluge depends on libffi, so that formula is a dependency here). A homebrew formula that uses Python creates a virtual environment under its location to use (conventionally named libexec) and installs any resources (e.g. python wheels). So, I have a directory like this under my Homebrew prefix:
$ cd /opt/homebrew/opt/deluge-meta
$ ls -Fl1
AUTHORS
bin/
CHANGELOG.md
INSTALL_RECEIPT.json
libexec/
LICENSE
README.md
share/
I can reproduce the error using the sample above with this formula's python (which depends on libffi and has pyopenssl installed):
$ /opt/homebrew/opt/deluge-meta/libexec/bin/python sample.py
Traceback (most recent call last):
File "/private/tmp/deluge/minimal_2.py", line 51, in <module>
get_context_factory()
File "/private/tmp/deluge/minimal_2.py", line 45, in get_context_factory
ctx = cert_options.getContext()
File "/private/tmp/deluge/minimal_2.py", line 21, in getContext
self._context = self._makeContext()
File "/private/tmp/deluge/minimal_2.py", line 37, in _makeContext
ctx.set_verify(verifyFlags, _verifyCallback)
File "/opt/homebrew/Cellar/deluge-meta/2.0.5/libexec/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1128, in set_verify
self._verify_helper = _VerifyHelper(callback)
File "/opt/homebrew/Cellar/deluge-meta/2.0.5/libexec/lib/python3.10/site-packages/OpenSSL/SSL.py", line 359, in __init__
self.callback = _ffi.callback(
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
However, I do not see the error with homebrew's python:
$ /opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -m pip install pyopenssl
$ /opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9 sample.py
sample worked!
Or with virtual environments created with homebrew's or pyenv's python
# homebrew python 3.9
$ /opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -m venv venv
# homebrew python 3.10
$ /opt/homebrew/Cellar/[email protected]/3.10.4/bin/python3 -m venv venv
# python 3.9 (pyenv)
$ /Users/amar1729/.pyenv/shims/python3 -m venv venv
# system python 3.8 does not work - pip installing cryptography fails due to missing openssl header
Using venvs created with any of these pythons (and pip installing pyopenssl) results in a working sample:
# install deps and run:
$ venv/bin/pip install pyopenssl
$ venv/bin/python sample.py
sample worked!
If you would like to reproduce this without installing my tap (understandable) I've created a simple formula definition which you can use:
# you never have to push this tap upstream, `brew` will use the local file just fine
$ brew tap-new arbitrary-username/openssl-env
$ vim $(brew --prefix)/Library/Taps/arbitrary-username/openssl-env/openssl-env.rb # paste the content of the next code block
$ brew install arbitrary-username/openssl-env/openssl-env
$ /opt/homebrew/opt/openssl-env/libexec/bin/python sample.py
<should fail>
openssl-env.rb
class OpensslEnv < Formula
include Language::Python::Virtualenv
desc "Meta package for pyopenssl testing"
homepage "https://github.com/pyca/pyopenssl"
url "https://github.com/pyca/pyopenssl/archive/refs/tags/22.0.0.tar.gz"
sha256 "4e92f6c2f08a8d1f0d9695335037a3d50ef8f58cd326514b89104acb9abb2838"
depends_on "rust" => :build
depends_on "libffi"
depends_on "[email protected]"
resource "cffi" do
url "https://files.pythonhosted.org/packages/00/9e/92de7e1217ccc3d5f352ba21e52398372525765b2e0c4530e6eb2ba9282a/cffi-1.15.0.tar.gz"
sha256 "920f0d66a896c2d99f0adbb391f990a84091179542c205fa53ce5787aff87954"
end
resource "cryptography" do
url "https://files.pythonhosted.org/packages/51/05/bb2b681f6a77276fc423d04187c39dafdb65b799c8d87b62ca82659f9ead/cryptography-37.0.2.tar.gz"
sha256 "f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
end
resource "pycparser" do
url "https://files.pythonhosted.org/packages/5e/0b/95d387f5f4433cb0f53ff7ad859bd2c6051051cebbb564f139a999ab46de/pycparser-2.21.tar.gz"
sha256 "e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"
end
resource "pyOpenSSL" do
url "https://files.pythonhosted.org/packages/35/d3/d6a9610f19d943e198df502ae660c6b5acf84cc3bc421a2aa3c0fb6b21d1/pyOpenSSL-22.0.0.tar.gz"
sha256 "660b1b1425aac4a1bea1d94168a85d99f0b3144c869dd4390d27629d0087f1bf"
end
def install
virtualenv_install_with_resources
end
test do
system "true"
end
end
Note the line depends_on "libffi". I've found it is that line in particular that causes this failure from a formula's python. If you remove it (or change it to uses_from_macos "libffi", which uses the system version of libffi), I see that sample.py running perfectly fine. While uses_from_macos is an acceptable solution for formulae that depend directly on libffi, it is infeasible for formulae that recursively depend on it and use the homebrew version (e.g. my deluge formula, which depends on several deps that depend on cairo > glib > libffi).
I've tried various invocations of codesign, as suggested by the cffi docs and documented in my repo's issue, but even signing the python symlinks or homebrew libffi/lib/libffi.dylib with the appropriate entitlements doesn't make MacOS let a formula-bundled python run an insecure script.
