spark-llap icon indicating copy to clipboard operation
spark-llap copied to clipboard

need to make branch-2.4.0 compatible with spark 2.4.0

Open abh1sh2k opened this issue 6 years ago • 2 comments

Currently method like executeQuery does not work when running this library with spark 2.4.0 because in spark 2.4.0 some classes have been removed and some classes have been renamed

  1. DataReaderFactory have been renamed to InputPartition 2)DataReader have been renamed to InputPartitionReader 3)SupportsWriteInternalRow have been removed

Exception while running executeQuery

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/reader/DataReaderFactory at com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector.getDataSourceReader(HiveWarehouseConnector.java:64) at com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector.createReader(HiveWarehouseConnector.java:37) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$SourceHelpers.createReader(DataSourceV2Relation.scala:155) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:172) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:204) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:62) ... 49 elided Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.reader.DataReaderFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 56 more

abh1sh2k avatar Jul 07 '19 08:07 abh1sh2k

I have created a pr https://github.com/abh1sh2k/spark-llap/pull/1/files here . if anybody can look . It is working fine for spark 2.4. Would love to make pr in this repo? currently it seems I don't have permission to even push to a branch .

abh1sh2k avatar Jul 10 '19 07:07 abh1sh2k

I created the PR #282 based on @abh1sh2k's branch mentioned above.

carloslima avatar Mar 23 '20 12:03 carloslima