this NDArray implementation does not currently support this operation

[leezea]

trafficstars

Description

when i run the jupyter/onnxruntime/machine_learning_with_ONNXRuntime in spark, if i run it in spark_local modle with command "spark-submit --class org.example.pro1.Test iris_test.jar" that is ok, but when i run it in spark_yarn model with commad "spark-submit --queue tst --master yarn --executor-memory 20g --driver-memory 20g --executor-cores 4 --num-executors 4 --class org.example.pro1.Test iris_test.jar" ,the error appares like this "Caused by: java.lang.UnsupportedOperationException: This NDArray implementation does not currently support this operation" why this appends? thanks for your reply! my pom.xml like this : <spark.version>2.2.0</spark.version> <scala.version>2.11</scala.version> <maven.version>3.6.3</maven.version> <java.version>1.8</java.version>

ai.djl.hadoop hadoop 0.12.0

<!--scala-->
<dependency>
  <groupId>org.scala-lang</groupId>
  <artifactId>scala-library</artifactId>
  <version>2.11.8</version>
  <scope>compile</scope>
</dependency>
<dependency>
  <groupId>org.scalanlp</groupId>
  <artifactId>breeze_2.11</artifactId>
  <version>0.12</version>
</dependency>
<dependency>
  <groupId>org.scalanlp</groupId>
  <artifactId>breeze-viz_2.11</artifactId>
  <version>0.12</version>
</dependency>

<!--djl-->
<dependency>
  <groupId>ai.djl</groupId>
  <artifactId>api</artifactId>
  <version>0.12.0</version>
</dependency>
<dependency>
  <groupId>ai.djl.onnxruntime</groupId>
  <artifactId>onnxruntime-engine</artifactId>
  <version>0.12.0</version>
</dependency>
<dependency>
  <groupId>com.microsoft.onnxruntime</groupId>
  <artifactId>onnxruntime</artifactId>
  <version>1.4.0</version>
</dependency>

<!--djl-pytorch-->
<dependency>
  <groupId>ai.djl.pytorch</groupId>
  <artifactId>pytorch-engine</artifactId>
  <version>0.12.0</version>
</dependency>
<dependency>
  <groupId>ai.djl.pytorch</groupId>
  <artifactId>pytorch-model-zoo</artifactId>
  <version>0.12.0</version>
</dependency>
<dependency>
  <groupId>ai.djl.pytorch</groupId>
  <artifactId>pytorch-native-auto</artifactId>
  <version>1.8.1</version>
</dependency>
<dependency>
  <groupId>net.java.dev.jna</groupId>
  <artifactId>jna</artifactId>
  <version>5.3.0</version>
</dependency>

...

[frankfliu]

@leezea Can you try 0.14.0?

OnnxRuntime itself doesn't support NDArray operations, it rely on PyTorch engine to do it. We made some changes in HyBird engine implementation in 0.13.0. Please see if the issue is fixed in latest version.

[leezea]

thanks for your reply! my local system is ubuntu16.04, if i using the version of djl like 13 or 14, there will appare one error like " /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.22' not found" ,i can install libstdc++ to fix this bug, but there still another error like "/lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.27' not found",which looks like as if it not support in ubuntu16? so it means i can't using 13 or 14 on ubuntu16.04, although i can change an other high version system like ubuntu18,but the yarn system is ubuntu16, which means i can't use djl on it?

[frankfliu]

@leezea Your error is linux glibc version issue, DJL 0.13/0.14 upgrade pytorch to 1.9.1, which doesn't support ubuntu 16.04 any more.

1. ubuntu 16.04 is end of life already, many of the engine dropped 16.04 support already, we don't recommend to use 16.04 any more
2. We have pre-cxx11 native package that can run on lower version of glibc system, see: https://docs.djl.ai/engines/pytorch/pytorch-engine/index.html#for-pre-cxx11-build

[siddvenk]

Closing due to inactivity. Please reopen if you still have questions.