dagster
dagster copied to clipboard
Published
20 hours ago •
dagster-io
dagster-io
[dagit] Dagit does not run on M1 (Apple Silicon)
Summary
I was going through the Getting Started docs for Dagster and was following along on a newer Macbook using an M1 chip. When I tried to start Dagit, I ran into an error, and I'm not able to figure out a quick workaround.
I think the issue might be with an upstream dependency, but I wanted to report it here first, in case I was mistaken.
Reproduction
- Install Dagster with
pip install dagster. - Create
hello_world.pyas described in the getting started docs. - Install Dagit with
pip install dagit. - Run
dagit -f hello_world.py
Expected behavior
The Dagit command does not output an error, and localhost:3000 hosts an instance of the Dagit UI.
Actual behavior
The Dagit command fails with this error:
Traceback (most recent call last):
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/bin/dagit", line 5, in <module>
from dagit.cli import main
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagit/__init__.py", line 1, in <module>
from dagster.core.utils import check_dagster_package_version
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/__init__.py", line 175, in <module>
from dagster.core.launcher import DefaultRunLauncher
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/launcher/__init__.py", line 1, in <module>
from .base import (
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/launcher/base.py", line 8, in <module>
from dagster.core.workspace.workspace import IWorkspace
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/workspace/__init__.py", line 1, in <module>
from .context import IWorkspaceProcessContext, WorkspaceProcessContext
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/workspace/context.py", line 13, in <module>
from dagster.core.host_representation import (
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/host_representation/__init__.py", line 9, in <module>
from .external import (
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/host_representation/external.py", line 24, in <module>
from .handle import JobHandle, PartitionSetHandle, PipelineHandle, RepositoryHandle
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/host_representation/handle.py", line 4, in <module>
from dagster.core.host_representation.origin import ExternalRepositoryOrigin
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dagster/core/host_representation/origin.py", line 8, in <module>
import grpc
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/grpc/__init__.py", line 22, in <module>
from grpc import _compression
File "/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/grpc/_compression.py", line 15, in <module>
from grpc._cython import cygrpc
ImportError: dlopen(/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): tried: '/Users/praneetsahgal/.pyenv/versions/3.10.0/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cygrpc.cpython-310-darwin.so' (no such file), '/usr/lib/cygrpc.cpython-310-darwin.so' (no such file)
Additional Info about Your Environment
- Device: 2021 16-inch Macbook Pro (running an M1 chip)
- Python version: 3.10.0
- Pip version: 21.3.1
- I was prompted to install Rosetta earlier, so I think I have it installed.
Message from the maintainers:
Impacted by this bug? Give it a 👍. We factor engagement into prioritization.
Hi @psahgal - while we sort this out, this issue seems related - dagster uses the grpc library which seems to have some m1-specific problems: https://github.com/grpc/grpc/issues/25082
Do you know what version of grpcio you have installed?
I am on version 1.43.0 of gRPC. I didn't explicitly install it; I think it was a dependency for either Dagster or Dagit.
We in Scale AI are also affected by this bug. Any expediency that can be put into getting this resolved would be great.
The workaround in the grpc issue mentioned above worked for me
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
pip install dagit
dagit -f /tmp/repro.py
Hey folks - I work on the dagit UI and use an M1 MBP, I unfortunately don't remember what I had to do to get my env set up a couple months ago, but I'm using Python 3.9.7 (upgraded from 3.8.5 recently I think) and my grpc install actually looks like it's x86_64:
lipo /Users/bengotow/.pyenv/versions/3.8.5/envs/dagster2021/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so -detailed_info
input file /Users/bengotow/.pyenv/versions/3.8.5/envs/dagster2021/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so is not a fat file
Non-fat file: /Users/bengotow/.pyenv/versions/3.8.5/envs/dagster2021/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so is architecture: x86_64
I think I'm actually running all of Dagster in an x86_64 build of Python via MacOS Rosetta:
(dagster2021) bengotow@Bens-M1-MBP software_defined_assets % python3
Python 3.9.7 (default, Oct 22 2021, 13:39:39)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.architecture()
('64bit', '')
>>> platform.machine()
'x86_64'
Hope that helps a bit, if there's anything else I can provide I'm happy to!
The workaround in the grpc issue mentioned above worked for me
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install dagit dagit -f /tmp/repro.py
did you run that with the terminal in rosetta 2 ?
i tried these steps here, but no success.
did you run that with the terminal in rosetta 2 ?
This was in a non-rosetta terminal, and I checked the python processes in Activity Monitor to ensure they were Kind Apple.
Using python 3.9.x and installing grpc with conda, solved the problem to me
https://github.com/grpc/grpc/issues/25082#issuecomment-878271247
conda install grpcio
I think this was the same issue for me... for python 3.10 anyway,
the solution for python 3.10 was to uninstall grpcio and reinstall without the wheel using the instruction from SaneBow here: https://github.com/grpc/grpc/issues/28387#issuecomment-1016166913
Quick solution for users is to force pip to build the wheel:
pip install --no-binary :all: grpcio --ignore-installed
This info is in the linked gRPC thread, but just so people don't have to click through-- the problem is that the grpcio package is releasing wheels for Python 3.10 that are incorrectly labeled as universal binaries when in fact they only support x86-64. So when you install dagit on Python 3.10 on an M1 mac, pip resolves the grpcio against this mislabeled binary, but then finds it can't actually use it.
I did a little research and unfortunately I don't think there's any way to have setuptools specify a --no-binary installation for a specific package (which would fix the issue). So for now the best we can do is provide a workaround and wait till the grpc maintainers fix their wheels.
I'm running into this issue too. If I do conda install grpcio after pip install then dagit will work. However, I ran into trouble when upgrading dagster and had to nuke my conda environment and start all over to get it working again.
Python 3.9.12
I solved the issue following this commands:
Try this - it worked on my M1 Max MacBook Pro:
python3 -m venv dagster
source dagster/bin/activate
pip3 install dagster dagit
from: https://apple.stackexchange.com/questions/438915/install-dagster-on-m1-macbook
I am still having this error months later and have spent 3 days trying to resolve it with no success so far. Dagit should bump the version requirement for grpcio to use the latest version (1.50) which seems to work on aarch64 instead of using 1.47 or 1.43. Are there any plans to do this? It seems odd that dagit still does not run on M1 in Nov 2022.
It seems one cause of this error still effecting some people is not having pip up to date. Make sure you have the most up to date pip (pip install -U pip) in your environment before you install dagster / dagit .
I get this error. It worked fine before (June 2022) on M1. Get the error by simply installing pip install dagster (and grpcio).
pip should be on the latest:
❯ python --version
Python 3.9.15
❯ pip --version
pip 22.3.1 from /Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/pip (python 3.9)
Anything more to consider 🤔. (I recently switched to asdf for setting up python versions, that shouldn't cause the error?)
Tail of grpcio error:
...
envs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/cares/cares/src/lib/ares__close_sockets.c -o python_build/temp.macosx-12.6-arm64-3.9/third_party/cares/cares/src/lib/ares__close_sockets.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
In file included from third_party/cares/cares/src/lib/ares__close_sockets.c:17:
In file included from third_party/cares/cares/src/lib/ares_setup.h:32:
third_party/cares/config_darwin/ares_config.h:319:9: warning: 'HAVE_UNISTD_H' macro redefined [-Wmacro-redefined]
#define HAVE_UNISTD_H
^
<command line>:11:9: note: previous definition is here
#define HAVE_UNISTD_H 1
^
1 warning generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/parse.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/parse.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/abseil-cpp/absl/strings/internal/cordz_info.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/abseil-cpp/absl/strings/internal/cordz_info.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c src/core/lib/transport/handshaker_registry.cc -o python_build/temp.macosx-12.6-arm64-3.9/src/core/lib/transport/handshaker_registry.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
third_party/re2/re2/parse.cc:1592:15: warning: code will never be executed [-Wunreachable-code]
LOG(DFATAL) << "Not reached in ParseEscape.";
^~
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c src/core/lib/transport/http_connect_handshaker.cc -o python_build/temp.macosx-12.6-arm64-3.9/src/core/lib/transport/http_connect_handshaker.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
1 warning generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/perl_groups.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/perl_groups.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/prefilter.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/prefilter.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c src/core/lib/transport/metadata_batch.cc -o python_build/temp.macosx-12.6-arm64-3.9/src/core/lib/transport/metadata_batch.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/prefilter_tree.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/prefilter_tree.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/prog.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/prog.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
third_party/re2/re2/prog.cc:514:16: warning: code will never be executed [-Wunreachable-code]
LOG(ERROR) << "Using trivial bytemap.";
^~
third_party/re2/re2/prog.cc:513:7: note: silence by adding parentheses to mark code as explicitly dead
if (0) { // For debugging, use trivial bytemap.
^
/* DISABLES CODE */ ( )
third_party/re2/re2/prog.cc:613:7: warning: variable 'total' set but not used [-Wunused-but-set-variable]
int total = 0;
^
2 warnings generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/re2.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/re2.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX="Python" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX="1.47.2" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c third_party/re2/re2/regexp.cc -o python_build/temp.macosx-12.6-arm64-3.9/third_party/re2/re2/regexp.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
creating None
creating None/var
creating None/var/folders
creating None/var/folders/5p
creating None/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn
creating None/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T
creating None/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/tmpze4skkn9
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/sspaeti/.venvs/dagster-test/include -I/Users/sspaeti/.asdf/installs/python/3.9.15/include/python3.9 -c /var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/tmpze4skkn9/a.c -o None/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/tmpze4skkn9/a.o
Traceback (most recent call last):
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] +
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
_classic_spawn(self, command)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/spawn.py", line 91, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/commands.py", line 280, in build_extensions
build_ext.build_ext.build_extensions(self)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
_build_ext.build_extension(self, ext)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
objects = self.compiler.compile(sources,
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/commands.py", line 263, in new_compile
return old_compile(obj, src, ext, cc_args, extra_postargs,
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/setup.py", line 541, in <module>
setuptools.setup(
File "/Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/install.py", line 546, in run
self.run_command('build')
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/commands.py", line 284, in build_extensions
raise CommandError(
commands.CommandError: Failed `build_ext` step:
Traceback (most recent call last):
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] +
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
_classic_spawn(self, command)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/spawn.py", line 91, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/commands.py", line 280, in build_extensions
build_ext.build_ext.build_extensions(self)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/Users/sspaeti/.venvs/dagster-test/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
_build_ext.build_extension(self, ext)
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
objects = self.compiler.compile(sources,
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/private/var/folders/5p/my46d4_n1jx149py_dk9jzyc0000gn/T/pip-install-anh76k9f/grpcio_5d815da6502e4155a003b5dbbb886be3/src/python/grpcio/commands.py", line 263, in new_compile
return old_compile(obj, src, ext, cc_args, extra_postargs,
File "/Users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> grpcio
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
OK, I finally solved the issue for me. I spend hours on it and as I installed it forth and back it's hard to say what in the end solved the problem. But still, I want to give some pointers to folks who have similar issues. My problem was deeper that asdf was probably having a wonky python installation or that this installation wasn't properly set up with arch=arm64/
I believe at the end solved it by completely removing asdf as explained here. And reinstall with these additional settings:
brew update && upgrade
brew install llvm
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
export PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@3
Now I could properly install any python version without errors such as
asdf install python 3.9.15
asdf global python 3.9.15
I could then successfully install the problematic grpcio and also run dagster/dagit without below error that I got the strange error in the meantime.
rm -r ~/.venvs/dagster-test && python -m venv ~/.venvs/dagster-test && source ~/.venvs//dagster-test/bin/activate && pip install --upgrade pip && pip install dagster dagit
The error I got after I managed to install grpcio when starting dagit or dagster:
importerror: dlopen(/users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): tried: '/users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/system/volumes/preboot/cryptexes/os/users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so' (no such file), '/users/sspaeti/.asdf/installs/python/3.9.15/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
I'm seeing the same issue, none of the above solutions helped
My setup: Mac M2 Using remote development via docker desktop
vscode ➜ /workspaces/a-api-integration (setup ✗) $ pip install dagit
Defaulting to user installation because normal site-packages is not writeable
Collecting dagit
Using cached dagit-1.1.9-py3-none-any.whl (6.3 MB)
Collecting uvicorn[standard]
Using cached uvicorn-0.20.0-py3-none-any.whl (56 kB)
Collecting dagster==1.1.9
Using cached dagster-1.1.9-py3-none-any.whl (1.1 MB)
Collecting click<9.0,>=7.0
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting watchdog>=0.8.3
Using cached watchdog-2.2.1-py3-none-manylinux2014_aarch64.whl (78 kB)
Collecting PyYAML
Using cached PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (731 kB)
Collecting requests
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting starlette
Using cached starlette-0.23.1-py3-none-any.whl (64 kB)
Collecting dagster-graphql==1.1.9
Using cached dagster_graphql-1.1.9-py3-none-any.whl (134 kB)
Collecting tabulate
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting universal-pathlib
Downloading universal_pathlib-0.0.21-py3-none-any.whl (25 kB)
Collecting Jinja2
Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 888.7 kB/s eta 0:00:00
Collecting sqlalchemy>=1.0
Downloading SQLAlchemy-1.4.46-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 935.1 kB/s eta 0:00:00
Collecting packaging<22,>=20.9
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pydantic
Downloading pydantic-1.10.4-py3-none-any.whl (155 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 KB 979.7 kB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from dagster==1.1.9->dagit) (58.1.0)
Collecting protobuf<4,>=3.13.0
Using cached protobuf-3.20.3-cp39-cp39-manylinux2014_aarch64.whl (918 kB)
Collecting typing-extensions>=4.0.1
Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting coloredlogs<=14.0,>=6.1
Downloading coloredlogs-14.0-py2.py3-none-any.whl (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.9/43.9 KB 5.2 MB/s eta 0:00:00
Collecting python-dotenv
Using cached python_dotenv-0.21.0-py3-none-any.whl (18 kB)
Collecting tqdm
Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting alembic!=1.6.3,!=1.7.0,>=1.2.1
Using cached alembic-1.9.1-py3-none-any.whl (210 kB)
Collecting grpcio<1.48.1,>=1.32.0
Downloading grpcio-1.47.2-cp39-cp39-manylinux_2_17_aarch64.whl (62.2 MB)
━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/62.2 MB 953.7 kB/s eta 0:01:00
ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
yield
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
data = self.__fp.read(amt)
File "/usr/local/lib/python3.9/http/client.py", line 463, in read
n = self.readinto(b)
File "/usr/local/lib/python3.9/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 339, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
return bool(self._sequence)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
return any(self)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 215, in _make_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 288, in __init__
super().__init__(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
self.dist = self._prepare()
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
dist = self._prepare_distribution()
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 532, in _prepare_linked_requirement
local_file = unpack_url(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 214, in unpack_url
file = get_http_url(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in get_http_url
from_path, content_type = download(link, temp_dir.path)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/download.py", line 146, in __call__
for chunk in chunks:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/progress_bars.py", line 304, in _rich_progress_bar
for chunk in iterable:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
for chunk in response.raw.stream(
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
vscode ➜ /workspaces/a-api-integration (setup ✗) $
After a few hours of testing I was able to get it to work by changing to:
"image": "mcr.microsoft.com/devcontainers/python:0-3.10",
Hi all! Believe have a solution for this for now by providing our own wheel for grpcio that works for M1 machines, built from the grpcio 1.47.2 release with no changes. If you've been having trouble with this issue and wouldn't mind trying out one of the commands below, we'd appreciate to know whether or not they work for you.
On pip:
pip install dagster dagit --find-links=https://github.com/dagster-io/build-grpcio/wiki/Wheels
On poetry:
poetry source add grpcio https://github.com/dagster-io/build-grpcio/wiki/Wheels
poetry add dagster dagit
Hopefully soon grpcio will provide the wheels themselves, but in the interim we're hoping that this will help.
@gibsondan Thanks!! For some reason I cannot replicate the issue now.
I really do think I'm doing exactly what I did last time.
One reason might be. I'm using VS Code devcontainer, and Docker Desktop made an update that may fix this https://docs.docker.com/desktop/release-notes/#4160
I think grpcio is now providing the m1 wheels - however, dagster still pins to an older version for which these new builds are not yet available
grpcio just released a 1.47.5 release today that should install cleanly on M1 macs when installing Dagster - I'm going to close this out as a result but let us know if you are having any similar issues going forward.