connector-x
connector-x copied to clipboard
sfu-db
Building wheel for aarch platform
Releasing version for aarch in order to support Docker on M1 - related to #535
Hi @MatsMoll , have you tried to run this release script? I think we had some issues with this, so we comment it out (mentioned here but I kind of forgot the reason of failure).
Hi @MatsMoll any update regarding this issue when it will merge? Are any temporary fixes available?
I got focused on some other stuff and forgot to answer.
However, I have not tried to run the release script @wangxiaoying. As I am unsure how to test it locally. So I guess I mainly wanted to start a new discussion about this, and hopefully look into how to fix it. But I never got to the last part
We currently build from source using a multi-stage docker build and has been running on arch now for +- 3months without problem! Here we describe that multi-stage build. Perhaps a multi-stage github action could solve this problem?
@wangxiaoying how are you feeling about releasing this? Seems it would close quite a few issues: https://github.com/sfu-db/connector-x/issues/186 https://github.com/sfu-db/connector-x/issues/214 https://github.com/sfu-db/connector-x/issues/240 https://github.com/sfu-db/connector-x/issues/386 https://github.com/sfu-db/connector-x/issues/535 https://github.com/sfu-db/connector-x/pull/545
@wangxiaoying how are you feeling about releasing this? Seems it would close quite a few issues: #186 #214 #240 #386 #535 #545
Any update on this?
I got focused on some other stuff and forgot to answer.
However, I have not tried to run the release script @wangxiaoying. As I am unsure how to test it locally. So I guess I mainly wanted to start a new discussion about this, and hopefully look into how to fix it. But I never got to the last part
IC, I tried to test it in prerelease ci: https://github.com/sfu-db/connector-x/actions/runs/8014425468, but it seems like the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8014425468/job/21892997456#step:13:17
Any idea how to fix this issue?
@wangxiaoying can you please try upgrading maturin-action version?: https://github.com/sfu-db/connector-x/actions/runs/8014425468/workflow#L48
Currently the latest version is 1.41.0 but you are using 0.14.15. Maybe that issue has long been fixed?
@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?
@wangxiaoying can you please try upgrading maturin-action version?: https://github.com/sfu-db/connector-x/actions/runs/8014425468/workflow#L48
Currently the latest version is
1.41.0but you are using0.14.15. Maybe that issue has long been fixed?
I tried to upgrade maturin-action to the latest version, but it seems like the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20
@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?
Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20
@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?
Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20
It looks like Docker is not installed on the machine on which you're running this workflow.
@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?
Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20
It looks like Docker is not installed on the machine on which you're running this workflow.
We are running our release workflow using github-action and on github-hosted runners, which is why we need to compile it by specifying --target aarch64-unknown-linux-gnu.
The error here is still the same with the one I encountered back in this issue: https://github.com/sfu-db/connector-x/issues/240.
is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50
The default container for ubuntu-latest supposedly has docker installed
is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50
The default container for
ubuntu-latestsupposedly has docker installed
That's a good point. The previous container may be wrong. According to maturin's doc, we may need to use a different docker mentioned for aarch in the table. However, I tried https://ghcr.io/rust-cross/manylinux_2_28-cross:aarch64 with no luck. It got error on setup-poetry before reaching to maturin: https://github.com/sfu-db/connector-x/actions/runs/8091126113/job/22109709826#step:7:7
is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50 The default container for
ubuntu-latestsupposedly has docker installedThat's a good point. The previous container may be wrong. According to maturin's doc, we may need to use a different docker mentioned for aarch in the table. However, I tried
https://ghcr.io/rust-cross/manylinux_2_28-cross:aarch64with no luck. It got error onsetup-poetrybefore reaching to maturin: https://github.com/sfu-db/connector-x/actions/runs/8091126113/job/22109709826#step:7:7
what about this one: quay.io/pypa/manylinux_2_28_aarch64:latest?
quay.io/pypa/manylinux_2_28_aarch64:latest
It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19
quay.io/pypa/manylinux_2_28_aarch64:latest
It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19
but that's x86 not aarch64
Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.
quay.io/pypa/manylinux_2_28_aarch64:latest
It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19
but that's x86 not aarch64
Seems like this one cannot work on the github-hosted runner: https://github.com/sfu-db/connector-x/actions/runs/8105184340/job/22153117379#step:2:115
Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.
I think you don't need to build every time. Can you store the built wheel file to a place and fetch the wheel file to install everytime you start an instance?
Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.
I think you don't need to build every time. Can you store the built wheel file to a place and fetch the wheel file to install everytime you start an instance?
I'll store it in s3 and go from there. Thanks!
Seems like we still cannot find a way to do this in our current release workflow.
I am closing this PR for now but feel free to reopen it anytime.
@wangxiaoying there's roughly half a dozen issues in this repo requesting support for Arm Linux, so why close this issue if it's the root cause preventing that support?
What do you see as the work around? I currently am forced to use something besides connectorx when connecting to a db on Arm Linux, writing to a text file, then having polars read that text file.
there's roughly half a dozen issues in this repo requesting support for Arm Linux, so why close this issue if it's the root cause preventing that support?
This is why the issues remain open, looking for help. But the change made by this PR does not really solve the problem and we currently cannot find a way to make it work. Therefore I closed this PR. But again, please feel free to open a PR if you find a way to making the release workflow support Arm Linux. I think you can fork the repo and test the gihub actions locally.
What do you see as the work around?
For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: https://github.com/sfu-db/connector-x/pull/545#issuecomment-1973226403 .
For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: #545 (comment) .
I put together a simple configurable Dockerfile build script for this, for anyone interested in easily building a wheel file for Linux ARM64:
https://gist.github.com/martijnvdwoude/39dcb7f9d8be2b2f3c2e224f32b800b7
For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: #545 (comment) .
I put together a simple configurable Dockerfile build script for this, for anyone interested in easily building a wheel file for Linux ARM64:
https://gist.github.com/martijnvdwoude/39dcb7f9d8be2b2f3c2e224f32b800b7
Thanks for putting this together! Rather than spending hours building Python with a makefile, consider checking Docker for a prebuilt image with the required Python version. I opted for the image arm64v8/python:3.9.19-bookworm instead of the one you're using. Here's the Dockerfile with the change I made.
@bfgdigital I think that's a good point. Probably within the next 3 years Microsoft will have better support for ARM containers, so just as a stop gap why not release the latest connector-x arm linux builds? The chances of this library changing significantly are pretty low, and if needed one could build it themselves.
Would help a lot of people just releasing something to Pypi