flink-docker icon indicating copy to clipboard operation
flink-docker copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-14241][test]Add arm64 support for docker e2e test -- work in progress

Open wangxiyuan opened this issue 4 years ago • 9 comments

The docker image openjdk:8-jre only works for amd64.

When running test on arm64, use arm64v8/openjdk:8-jre instead.

wangxiyuan avatar May 18 '20 08:05 wangxiyuan

@zentol can you take a look? Thanks very much

wangxiyuan avatar Jun 10 '20 06:06 wangxiyuan

@rmetzger could you please take a look? This block arm64 test for a while. Thanks.

wangxiyuan avatar Jun 15 '20 02:06 wangxiyuan

I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93

also, the architecture seems to depend on the architecture of the machine executing the generate.sh script. But this script only generates the Dockerfile and metadata file, not the actual docker image.

rmetzger avatar Sep 08 '20 13:09 rmetzger

I'm not sure if this PR works as-is. The updated generator doesn't seem to set the architecture flag correctly: https://github.com/apache/flink-docker/pull/23/files#diff-0c5f825c10334c2eff0a90687f513c69R93

also, the architecture seems to depend on the architecture of the machine executing the generate.sh script. But this script only generates the Dockerfile and metadata file, not the actual docker image.

Yeah, many docker images support multi arch now, for example openjdk:11-jre 1. For this kind of image, we don't need to change any code here of cause. Different machine use different image automatically.

But there are still some images don't support arm64, for example, the image openjdk:8-jre2 which the e2e test uses. For this kind of image, we should use the image from another official repo arm64v83.

Now openlab runs flink e2e test on arm64 with my forked flink-docker repo.4 And the test runs well5

P.S. In openlab test, the swr.ap-southeast-3.myhuaweicloud.com/openlab/arm64v8 is just a mirror for arm64v8 to speed up the test.

wangxiyuan avatar Sep 09 '20 01:09 wangxiyuan

I'm sorry that I have to say it again, but I don't think the current approach is correct.

The generate.sh script generates a Dockerfile and metadata file, then this file is used to create a stackbrew file, which is committed here: https://github.com/docker-library/official-images/blob/master/library/flink. Then DockerHub does some magic to turn this into the official images.

If we want to add support for arm64 to the official Flink docker images, we probably need to generate an additional Dockerfile and metadata file with an updated FROM, and a correct release.metadata file.

I believe that your change works on ARM64 with the Flink e2e tests, but this repository is mostly used for building the official Flink docker images, and we need to make sure we are not breaking them with this change.

rmetzger avatar Sep 09 '20 06:09 rmetzger

Oh, I see. I didn't realize that this is used for offcial image build as well.

Yeah, If so, this PR is not correct. I'll find some way to only apply for e2e test. Thanks!!

wangxiyuan avatar Sep 09 '20 07:09 wangxiyuan

Great! Thanks!

rmetzger avatar Sep 09 '20 07:09 rmetzger

RocksDB is not currently being built for ARM. We had to manually compile it and override it in the classpath. Kinda hacky but it's the only solution we have until Flink supports it officially.

ghost avatar May 31 '21 13:05 ghost

The latest frocksdb version Flink is using (https://search.maven.org/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0/jar) supports Linux aarch64, so you should be able to run Flink on Docker on an M1 mac book (because Docker for Mac is effectively running in a (arm) Linux VM)

rmetzger avatar Jan 17 '22 12:01 rmetzger