SNOW-587514: Aarch64 wheels not published in pypi

[tbbharaj]

What is the current behavior?

pypi.org missing aarch64 wheels. This is in continuation to closed PR. I see github workflows have been updated to build aarch64-wheels

What is the desired behavior?

Get aarch64 wheels published to pypi.org

How would this improve snowflake-connector-python?

This will enable extending snowflake-connector python wheels support to arm64 platforms

References, Other Background

Github workflows are enabled to build aarch64 wheels: https://github.com/snowflakedb/snowflake-connector-python/blob/main/.github/workflows/build_test.yml#L92

[karakanb]

I am experiencing this issue in an indirect way if I understand correctly: running an arm64 Docker build for an Airflow image on an M1 Mac, due to the lack of the arm64 packages on pypi, pip is trying to build the package, but it doesn't have enough permissions to do that.

This is a sample Dockerfile to reproduce the issue":

FROM apache/airflow:2.3.0-python3.8

RUN pip install apache-airflow-providers-snowflake

This is how I build the image:

docker buildx build --platform linux/arm64  -t airflow-reproduction:arm64 .

The logs are long, but the gist of it seems to be a permission error:

#0 18.07       running build_ext
#0 18.07       building 'Cryptodome.Hash._MD2' extension
#0 18.07       creating build/temp.linux-aarch64-3.8
#0 18.07       creating build/temp.linux-aarch64-3.8/src
#0 18.07       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPYCRYPTO_LITTLE_ENDIAN -DSYS_BITS=64 -DLTC_NO_ASM -Isrc/ -I/usr/local/include/python3.8 -c src/MD2.c -o build/temp.linux-aarch64-3.8/src/MD2.o
#0 18.07       unable to execute 'gcc': Permission denied
#0 18.07       error: command 'gcc' failed with exit status 1
#0 18.07       [end of output]
#0 18.07   
#0 18.07   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 18.07   ERROR: Failed building wheel for pycryptodomex
#0 18.07   Running setup.py clean for pycryptodomex
#0 18.47 Failed to build snowflake-connector-python pycryptodomex
#0 18.47 ERROR: Could not build wheels for snowflake-connector-python, which is required to install pyproject.toml-based projects

This seems to occur because the base airflow image doesn't allow installing pip dependencies as root, but at the same time it seems like the build process wouldn't be necessary at all if the aarch64 builds were on PyPi already, if I understand correctly.

I can work around this, but I wanted to report this as a reproduction case here as well.

[toughrogrammer]

Hi. I got same error.

Below is an error.

FROM apache/airflow:2.3.2

USER root
RUN apt-get update --allow-unauthenticated --allow-insecure-repositories && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential libsasl2-dev libkrb5-dev

USER airflow
RUN pip install snowflake-connector-python

Below is not an error.

FROM apache/airflow:2.3.2

USER root
RUN apt-get update --allow-unauthenticated --allow-insecure-repositories && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential libsasl2-dev libkrb5-dev
RUN sudo -u airflow pip install snowflake-connector-python

[potiuk]

You have full information about extending/customizing airflow image with dependencies requiring build essentials, including examples in https://airflow.apache.org/docs/docker-stack/build.html

Start here https://airflow.apache.org/docs/docker-stack/build.html#extending-the-image for the build-essential requiring dependencies.

[sfc-gh-aling]

aarch64 wheels are now available on the pypi with the latest release of the python connector. I'm closing the issue but feel free to reach out if you're still encountering the issue.