spark-llap icon indicating copy to clipboard operation
spark-llap copied to clipboard
verified publisher icon hortonworks-spark

making hive warehouse connector compatible with spark 2.4

Open carloslima opened this issue 5 years ago • 4 comments

What changes were proposed in this pull request?

Currently method like executeQuery does not work when running this library with spark 2.4.0 because in spark 2.4.0 some classes have been removed and some classes have been renamed

DataReaderFactory have been renamed to InputPartition 2)DataReader have been renamed to InputPartitionReader 3)SupportsWriteInternalRow have been removed

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

carloslima avatar Mar 23 '20 12:03 carloslima

I did not write the patch, I'm just creating the PR based on #272

carloslima avatar Mar 23 '20 12:03 carloslima

I know this is old but: @carloslima did you test this and did this work for you with Spark 2.4?

lfrancke avatar Sep 07 '21 10:09 lfrancke

@lfrancke old, indeed :sweat_smile:

We opted to go a different route so I have no feedback to offer :-)

Please, note that I didn't write this patch. This is all @abh1sh2k work, the only thing I did was open this MR based on their fork that was mentioned in https://github.com/hortonworks-spark/spark-llap/issues/272.

Assuming @abh1sh2k still uses this patch, they may have something to share :-)

carloslima avatar Sep 12 '21 15:09 carloslima

Thanks anyway! I assumed as much :) The reason I was asking is because I tried it and while it compiles it fails with various errors at runtime.

lfrancke avatar Sep 16 '21 18:09 lfrancke