Orc properties for Hive Source and Sink

[hannesmiller]

trafficstars

I just want to confirm that we respect the following Hive properties outlined in the documentation:

• Hive: http://orc.apache.org/docs/hive-config.html

My only concern is that our Orc dialect for Hive Source and Sink delegates to the same code base for our native Orc Source and Sink and hence ignoring these table properties?

If this is the case we should automatically pick up these options from the Hive Metastore when using HiveSource and HiveSink, and support them natively when using the OrcSink and OrcSource?

[sksamuel]

We completely ignore them at the moment, but you're right we should honour them where appropriate. We can still use the same source and sink, but when the dialect delegates it would need to pass in a config object pre-populated.

[hannesmiller]

Cool! I am not sure whether we need to honour other Hive dialects such as Parquet and Avro?

I'll investigate this...