transpiler icon indicating copy to clipboard operation
transpiler copied to clipboard

NoSuchMethodErrors on cidr_match queries

Open mstair opened this issue 2 years ago • 2 comments

Getting following error on attempts to run cidr_match queries against following CREATE TABLE security_log (user_id STRING, remote_host_ip STRING). Cluster is 11.3 LTS ML (includes Apache Spark 3.3.0, Scala 2.12)

from dbl_transpiler import spl

display(spl("""
index=security_log remote_host_ip=109.177.0.0/16
"""))

...

Py4JJavaError: An error occurred while calling z:com.databricks.labs.transpiler.spl.Transpiler.toDataFrame.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.Cast$.apply$default$4()Z
	at org.apache.spark.sql.CidrMatch.aton(CidrMatch.scala:43)
	at org.apache.spark.sql.CidrMatch.ipAddress(CidrMatch.scala:24)
	at org.apache.spark.sql.CidrMatch.cidrMatch(CidrMatch.scala:22)

mstair avatar Apr 30 '23 22:04 mstair

For completeness, I tried to run on DB 12.0/12.2/13.0 clusters and got the same error.

mstair avatar May 23 '23 19:05 mstair

Same error on 13.2 cluster also. @nfx any update?

mstair avatar Jul 13 '23 14:07 mstair