Unable to run SQL queries

[drorata]

I have an EMR cluster running Livy and I'm trying to work from a local notebook using sparkmagic. In my notebook I have the following:

#### (cell separator)
%load_ext sparkmagic.magics

###
%manage_spark # where I add the endpoint and create a session

###
%%spark
df = spark.read.json("s3://mybucket/somedata/dt=2019-05-*")

###
%%spark
df.createOrReplaceTempView("data")

###
%%spark -c sql  # as per the example
SHOW TABLES

The last cell yields the following error:

An error was encountered:
u'java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;'
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
AnalysisException: u'java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;'

I tried to run pretty much the same cells directly on Zeppelin (which is installed and running on the EMR) and it all worked as expected.

[itamarst]

Hi,

What version Livy and SparkMagic?

[drorata]

SparkMagic version is 0.13.1 Livy version is 0.6.0

(Sorry for the delayed reply)