hadoop-hdfs-fsimage-exporter icon indicating copy to clipboard operation
hadoop-hdfs-fsimage-exporter copied to clipboard

Published 20 hours ago • verified publisher icon marcelmay

aarch64 image

Open tulid opened this issue 4 years ago • 7 comments

i want to run hbase on aarch64 env. but, i am lack of the image -- hadoop-hdfs-fsimage-exporter. can u support me the image file. thanks very much.

tulid avatar Nov 18 '20 01:11 tulid

It's difficult to test, as I do not have access to aarch64 environment.

But I modified the build so that you can choose a different base image.

Here's an example using an aarm64v8 openjdk base image:

 mvn clean install -DskipITs 
       -Ddocker.base.image=arm64v8/openjdk:11-jre-slim-buster
       -Dimage.name=marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8
       -Pdocker

marcelmay avatar Nov 22 '20 20:11 marcelmay

i conduct the command，but the error appear as blow： ... ... [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] 未与 -source 8 一起设置引导类路径 [WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[141,15] 找到原始类型: i o.prometheus.client.SimpleCollector 缺少泛型类io.prometheus.client.SimpleCollector<Child>的类型参数 [WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[145,15] 找到原始类型: i o.prometheus.client.SimpleCollector 缺少泛型类io.prometheus.client.SimpleCollector<Child>的类型参数 [WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[149,15] 找到原始类型: i o.prometheus.client.SimpleCollector 缺少泛型类io.prometheus.client.SimpleCollector<Child>的类型参数 [WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[158,15] 找到原始类型: i o.prometheus.client.SimpleCollector 缺少泛型类io.prometheus.client.SimpleCollector<Child>的类型参数 [INFO] 5 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[17,39] 程序包org.apache.h adoop.fs.permission不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[18,46] 程序包org.apache.h adoop.hdfs.server.namenode不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[491,53] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[499,58] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[504,56] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[302,57] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[327,62] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[349,60] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[303,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,17] 找不到符号 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,49] 无法访问org.apach e.hadoop.hdfs.server.namenode.FsImageProto.INodeSection 找不到org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeSection的类文件 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[328,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[329,17] 找不到符号 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[350,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[351,17] 找不到符号 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[492,38] 程序包FsImageProt o.INodeSection不存在 [INFO] 16 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.165 s [INFO] Finished at: 2020-11-23T13:37:36+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project fsimage-exporter: Compilat ion failure: Compilation failure: [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[17,39] 程序包org.apache.h adoop.fs.permission不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[18,46] 程序包org.apache.h adoop.hdfs.server.namenode不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[491,53] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[499,58] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[504,56] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[302,57] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[327,62] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[349,60] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[303,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,17] 找不到符号 [ERROR] 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,49] 无法访问org.apach e.hadoop.hdfs.server.namenode.FsImageProto.INodeSection [ERROR] 找不到org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeSection的类文件 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[328,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[329,17] 找不到符号 [ERROR] 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[350,42] 程序包FsImageProt o.INodeSection不存在 [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[351,17] 找不到符号 [ERROR] 符号: 类 PermissionStatus [ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[492,38] 程序包FsImageProt o.INodeSection不存在 [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

tulid avatar Nov 23 '20 05:11 tulid

That's strange. Can you provide the output of mvn -v ?

marcelmay avatar Nov 23 '20 06:11 marcelmay

mvn -vas blow: root@uos-test-1:~/jinquan/hadoop-hdfs-fsimage-exporter# mvn -v Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /usr/local/apache-maven-3.6.3 Java version: 11.0.4, vendor: Debian, runtime: /usr/lib/jvm/java-11-openjdk-arm64 Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "4.19.0-arm64-server", arch: "aarch64", family: "unix"

tulid avatar Nov 23 '20 06:11 tulid

The versions seem ok - the JVM 11 is not at latest patch level (eg 11.0.9+) but should work.

The compiler complains it can not find the Hadoop classes. It should have complained that it can not download these, but not during compilation.

Can you check the JARs, eg if INodeSection exists:

jar vft ~/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar|grep INodeSection.class

That should print something like

17813 Mon Jul 06 18:50:24 CEST 2020 org/apache/hadoop/hdfs/server/namenode/FsImageProto$INodeSection.class

I managed to run the (experimental) docker image marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8 locally via docker and qemu arm64 emulator, the exporter using the arm64 openjdk image seems to work:

docker run --rm -t -p 9709:9709 -v $PWD/src/test/resources:/fsimage-location \
    -e "JAVA_OPTS=-server -XX:+UseG1GC -Xmx1024m" \
    marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8

marcelmay avatar Nov 23 '20 21:11 marcelmay

i run the command as you said,but its result is an error as blow. can u build one image that tag is 1.2 for my emergency. thanks a lot

root@uos-test-1:~# jar vft ~/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar|grep INodeSection.class java.nio.file.NoSuchFileException: /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1763) at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225) at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:727) at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844) at java.base/java.util.zip.ZipFile.(ZipFile.java:247) at java.base/java.util.zip.ZipFile.(ZipFile.java:177) at java.base/java.util.zip.ZipFile.(ZipFile.java:148) at jdk.jartool/sun.tools.jar.Main.list(Main.java:1500) at jdk.jartool/sun.tools.jar.Main.run(Main.java:380) at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)

tulid avatar Nov 24 '20 05:11 tulid

Somehow there are no dependencies, there should have been more warnings when Maven tries to download these.

Can you try the docker image marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8 of current latest head, or do require exactly 1.2 version (latest is 1.4)?

docker pull marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8:1.5-SNAPSHOT

marcelmay avatar Nov 24 '20 19:11 marcelmay

Available on https://hub.docker.com/r/marcelmay/hadoop-hdfs-fsimage-exporter/tags

marcelmay avatar Oct 15 '22 11:10 marcelmay