Amogh Margoor

Hi @adiu19, Seems like some bug in broadcasting Jobconf. Can you set this config to really high value so that it bypasses this code path `spark.hiveAcid.parallel.partitioning.threshold` ? This should be...

@adiu19 yeah will take a look at it for next release. thanks for reporting.I may need some help with reproducing it if i cannot on our end.

@maheshk114 in that case i believe it should have failed without the flag being disabled too. but anyways a good point to consider. @adiu19 Can you check if you guys...

once sparkSession is created it should not allow to add spark.sql.extensions. I will fail like below: `scala> spark.conf.set("spark.sql.extensions", "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension") org.apache.spark.sql.AnalysisException: Cannot modify the value of a static config: spark.sql.extensions;` Which...

hey @yukiHan0517 ... were you able to resolve the issue ?

Couple of issues here: 1. One of the issues specific to ORC is that the reader currently understands the ACID format. So if we just do this: spark.read.format("orc").load(".../warehouse/acidtbl/delta_0000002_0000002_0000/bucket_00000") following exception...

Hi @vinay-kl, Looks like you are using the old jar. There was some major refactoring which happened since then. Can you build the latest master and use it's Jar ?...

hey @vinay-kl I guess it is because the conf being passed as argument to JobConf is null. That means the conf being passed at this location will be lost: https://github.com/qubole/spark-acid/blob/2e66f76b3a3c974a73f03d9e80d9c66a0475a552/src/main/scala/com/qubole/spark/hiveacid/reader/hive/HiveAcidReader.scala#L93...

@vinay-kl The original issue which you created said that this value is null: `The value is coming as null in this line---> https://github.com/qubole/spark-acid/blob/f445eeef4416ee27192905e0e69a43076db7b2b1/src/main/scala/com/qubole/spark/datasources/hiveacid/rdd/Hive3Rdd.scala#L138` So I assume this problem is solved...

But you just said conf is not null, so it cannot be same issue. Can you please paste the current stacktrace even if you think it is same ? Sent...