ap-airflow
ap-airflow copied to clipboard
astronomer
Support for Apple Silicon
Hello, do you plan support for images for Apple Silicon? Would be great to be able to run Astronomer on the new Macs! Right now the command astro dev start works but the containers (webserver, scheduler, database) refuse to work together.
Any news for this issue? Would it be possible to build the image for multi-arch using xbuild or any other solution? This seems to be the only way to fully support the new M1 chip for Mac.
Hi there @ilyanoskov and @GeraudK ! Thanks for reaching out, and apologies for the delayed response. Our team has been testing the Astronomer CLI and our ap-airflow images with Apple M1 machines and we've found that you can spin up your Airflow environment on astro dev start without a problem as long as you:
- Uninstall
pyarrow - Uninstall
apache-airflow-providers-google
Can you give that a shot and let us know if it works? You can also upgrade to Docker 3.3.2 - that addressed a longstanding bug with qemu. We're also exploring building a multi-arch image for long-term support, but we haven't gotten to that yet. For more info, check out this forum post.
@paolaperaza thank you for your answer! The problem arise when we try to add on things on top of your image (e.g. installing specific requirements such as tensorflow) and need an arm64 image for that but can't because your image isn't multi-arch. I'm glad you're contemplating the idea to support multi-arch.
@GeraudK Yes, unfortunately building too much on top of it might be a problem 😞 We'll very likely work towards getting that multi-arch image built.
Can I ask if you're an Astronomer customer? We typically take feature requests and what not via our Support Portal so they don't get lost and then track GH issues in a separate repo internally. Can I ask that you comment in the forum post above, and I'll make sure to update it as soon as we have multi-arch support? If you post, you'll get subscribed to the thread.
Thanks @GeraudK ! I'll go ahead and close out this issue from there.
@paolaperaza I have found a forum entry about this topic as well, here: https://forum.astronomer.io/t/astronomer-docker-container-images-for-apple-silicon/1038. Or you meant an Astronomer customer should file this request through the Support Portal?
Hey! is there any new information? I work with big data and yesterday I found out that due to emulation under m1, performance decreased at least 9 times (instead of performing a task in 10 minutes, it takes 90). This is work with tableauserverclient library
Same - any updates on the multi-platform images? I am getting a failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed error when running astro dev start. I was able to work around the issue by running DOCKER_DEFAULT_PLATFORM=linux/amd64 astro dev start ..... but this all be avoidable if proper aarch64 multi-platform images were provided in the first place so we could build off them.
ap-airflow has reach end of life at this point. It's successor, astro-runtime, does now have m1 support in some versions. You can find which versions in the architecture docs.
