Error with in unnamed module of loader ammonite.runtime.SpecialClassLoader @3e44f2a5 org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262)

[joesan]

trafficstars

I face a strange issue when trying out Almond and Ammonite. I'm using the Jupyter notebook extension using Almond and running a Scala kernel. I have the following content in one of my cell:

import org.apache.spark.sql._

val spark = {
  NotebookSparkSession.builder()
    .master("local[*]")
    .getOrCreate()
}

val df = spark.read
         .format("csv")
         .option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
         .option("header", "true")
         .option("multiLine", true)
         .option("mode", "DROPMALFORMED")
         .load("/path/to/file/data/file.csv")

df.printSchema()

And when run, this results in the error as below:

java.lang.IllegalAccessError: class org.apache.hadoop.mapreduce.lib.input.FileInputFormat tried to access method com.google.common.base.Stopwatch.<init>()V (org.apache.hadoop.mapreduce.lib.input.FileInputFormat and com.google.common.base.Stopwatch are in unnamed module of loader ammonite.runtime.SpecialClassLoader @3e44f2a5)
  org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262)
  org.apache.spark.input.StreamFileInputFormat.setMinPartitions(PortableDataStream.scala:51)
  org.apache.spark.rdd.BinaryFileRDD.getPartitions(BinaryFileRDD.scala:51)
  org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:253)
  scala.Option.getOrElse(Option.scala:189)
  org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:253)
  scala.Option.getOrElse(Option.scala:189)
  org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:253)
  scala.Option.getOrElse(Option.scala:189)
  org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
  org.apache.spark.rdd.RDD.$anonfun$take$1(RDD.scala:1343)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
  org.apache.spark.rdd.RDD.take(RDD.scala:1337)
  org.apache.spark.sql.execution.datasources.csv.MultiLineCSVDataSource$.infer(CSVDataSource.scala:328)
  org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:68)
  org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:63)
  org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$12(DataSource.scala:183)
  scala.Option.orElse(Option.scala:447)
  org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:180)
  org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
  org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
  org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
  org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
  ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:15)
  ammonite.$sess.cmd1$.<init>(cmd1.sc:7)
  ammonite.$sess.cmd1$.<clinit>(cmd1.sc:-1)

[alexarchambault]

That looks more like a JDK > 11 issue rather than an Ammonite or almond one. Do you get the same error with JDK 8?

[joesan]

So you mean I should downgrade to JDK 8?

[alexarchambault]

I think it should work, yes. What JDK version were you using?

[joesan]

java 11.0.10 2021-01-19 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)

[joesan]

Downgrading to Java 8 did not help either! I sitll get the same error!

[joesan]

Is there anything that could be done?