Linux ARM support

[thomasaarholt]

trafficstars

This is same as #237, but for linux.

Polars added linux arm support a few days ago (issue) (PR), which will also be wanted here, since Docker images running on M1 macs will be running arm architecture as well.

connectorx successfully installs in my arm docker image when I run pip install connectorx, but then fails when I try to import it:

import connectorx
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/fabrica/projects/forecast-compare/Untitled.ipynb Cell [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f666f7265636173742d636f6d70617265227d/fabrica/projects/forecast-compare/Untitled.ipynb#ch0000052vscode-remote?line=0)8' in <module>
----> 1[ import connectorx

File /usr/local/lib/python3.9/site-packages/connectorx/__init__.py:3, in <module>
      ]()[1](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=0)[ from typing import Optional, Tuple, Union, List, Dict, Any
----> ]()[3](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=2)[ from .connectorx_python import read_sql as _read_sql
      ]()[5](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=4)[ try:
      ]()[6](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=5)[     from importlib.metadata import version

ModuleNotFoundError: No module named 'connectorx.connectorx_python']()

If I require it to use the binary from pypi, then it does not find a match:

pip install -U connectorx --only-binary connectorx 
ERROR: Could not find a version that satisfies the requirement connectorx (from versions: none)
ERROR: No matching distribution found for connectorx

[wangxiaoying]

Hi @thomasaarholt , thanks for the issue. The source file on pypi is not ready yet so we have removed it from our new alpha version. We are working on it.

[messense]

Feel free to ping me if you need any help.

[wangxiaoying]

Hi @messense , can you please help to check this error (build wheel for arm linux): https://github.com/sfu-db/connector-x/runs/5337702956?check_suite_focus=true ? I have no idea how to fix it. The yaml file can be found here. Thank you!

[messense]

@wangxiaoying Try add manylinux: 2014 option to maturin-action.

[messense]

This is harder than we thought since you have system dependencies which aren't present in maturin-action's cross compilation docker images(they are based on Ubuntu 20.04).

https://github.com/sfu-db/connector-x/blob/fd7a61d910110310d521df66e90f0465942ccd2e/.github/workflows/release.yml#L46

And https://github.com/sfu-db/connector-x/blob/fd7a61d910110310d521df66e90f0465942ccd2e/.github/workflows/release.yml#L12

forces x86_64 so you may need to change to use QEMU and switch to the official aarch64 manylinux docker image instead, but it's gonna be at least 6x slow on CI.

It will be much easier if we're statically link system dependencies.

[messense]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

[wangxiaoying]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

Thank you so much! I will take a look at this.

[thomasaarholt]

Has anyone taken a look at this since February? I just met this again now - installing connectorx succeeds on aarch64, but it is not importable, resulting in the same error as in the first post.

[wangxiaoying]

Hi @thomasaarholt , we are now successfully supporting apple m1 with wheel files. But still having issue to cross compile for linux arm. Currently the only way to install connector-x on linux arm is to build it from source: https://sfu-db.github.io/connector-x/install.html#build-from-source-code