Unsupported OS/arch

[jinwenjie123]

What is the problem the feature request solves?

I am running a benchmark to evaluate the performance of Spark 3.4 with Comet in an AWS environment. I built the Comet JAR on my local machine (macOS), but encountered the following error: Driver WARN org.apache.comet.NativeBase Failed to load comet library java.lang.UnsupportedOperationException: Unsupported OS/arch, cannot find /org/apache/comet/linux/amd64/libcomet.so. Please try building from source. at org.apache.comet.NativeBase.bundleLoadLibrary(NativeBase.java:114) ~[comet-spark-spark3.4_2.13-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT] ... Comet extension is disabled due to an error while loading the native library, falling back to Spark.

It seems the Comet JAR I built on macOS is not compatible with the driver's environment. I assume I need to specify the target OS and architecture when building the JAR.

Could you provide guidance on how to specify the target environment when building the Comet JAR to solve this issue ?

Thanks!

Describe the potential solution

No response

Additional context

No response

[andygrove]

@jinwenjie123 I would recommend starting off by using a pre-built JAR which contains native binaries for multiple architectures.

https://datafusion.apache.org/comet/user-guide/installation.html#using-a-published-jar-file

[andygrove]

We use Docker images when building these jars. Our release process is documented at https://github.com/apache/datafusion-comet/tree/main/dev/release#readme

[jinwenjie123]

@jinwenjie123 I would recommend starting off by using a pre-built JAR which contains native binaries for multiple architectures.

https://datafusion.apache.org/comet/user-guide/installation.html#using-a-published-jar-file

I will give it a try. Many thanks !!

[jinwenjie123]

Hi @andygrove ,

Thanks for your previous suggestions.

However, when I use the precompiled JARs, I encounter a GLIBC_2.27 not found error.

I modified the Dockerfile to use amazon_linux:2 environment, which defaults to GLIBC_2.26. However, when compiling the libraries, I face the error: error: failed to run custom build command for ring v0.17.13`.

Could you suggest how to compile the Comet JAR in a way that's compatible with GLIBC_2.26 ? (Looks like this discussion relates to above issue)

Thank you so much !

[parthchandra]

You could try using an older version of ubuntu in the Dockerfile. You might have to go as far back as ubuntu 16.04 (see https://gist.github.com/wagenet/35adca1a032cec2999d47b6c40aa45b1) but there is no guarantee that the dependencies required by Comet will compile. You could also try zigbuild which can do a cross compilation for linux on macOS and also allows you to specify the glibc version. I haven't tried this but would be interested in knowing if it works.

[jinwenjie123]

@parthchandra

Looks like using ubuntu 16.04 dose not work since the dependencies will not compile successfully.

I tried to use the zigbuild, at least cd native && cargo zigbuild --target x86_64-unknown-linux-gnu.2.26 --release $(FEATURES_ARG) was executed successfully.

However, since I need to run it in amazonlinux2 environment, I try to do make core-amd64, but it looks like ./dev/deploy-file is missing.

[jinwenjie123]

I tried to run the benchmark for TPCH_Q1 query, but it looks like it dose not provide much improvement by adding comet. Maybe due to the following error:

org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning

org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark data type: MapType(StringType,StringType,true)

Is it because I missed to add some required libraries ? Thanks.

[parthchandra]

@parthchandra

Looks like using ubuntu 16.04 dose not work since the dependencies will not compile successfully.

I tried to use the zigbuild, at least cd native && cargo zigbuild --target x86_64-unknown-linux-gnu.2.26 --release $(FEATURES_ARG) was executed successfully.

However, since I need to run it in amazonlinux2 environment, I try to do make core-amd64, but it looks like ./dev/deploy-file is missing.

You should be able to use the library produced by zigbuild. After you build the comet library with zigbuild, mvn package should be able to build a jar with the library you just built. If the binary got built correctly, then it should be usable on amazonlinux2. (thank you for trying this out, this is new to me as well!).

[parthchandra]

org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning

org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark data type: MapType(StringType,StringType,true)

Is it because I missed to add some required libraries ?

No, it's not because of missing libraries. The messages say you're encountering something in the Spark plan that Comet does not support (RangePartitioning and the MapType specifically). This is odd because TPC-H does not have MapTypes. Do you have a modified version of the queries and data?

[jinwenjie123]

org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning org.apache.comet.serde.QueryPlanSerde Comet native execution is disabled due to: unsupported Spark data type: MapType(StringType,StringType,true) Is it because I missed to add some required libraries ?

No, it's not because of missing libraries. The messages say you're encountering something in the Spark plan that Comet does not support (RangePartitioning and the MapType specifically). This is odd because TPC-H does not have MapTypes. Do you have a modified version of the queries and data?

Hi @parthchandra , I checked the process, and it seems that the unsupported Maptype is introduced by the Delta conversion process added by our benchmark. After disabling Delta, the unsupported Maytype is no longer appeared but still get the Comet native execution is disabled due to: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning for using OSS 3.4.3.

And for TPCH queries, looks like Comet does not show significant performance advantages. For instance, the TPCH_Q1, I just saw around 10% improvement.

[parthchandra]

I suspect there is some more stuff your benchmark introduced. If Comet gets disabled because of missing support, you're naturally unlikely to see performance gains.