bigflow icon indicating copy to clipboard operation
bigflow copied to clipboard

Published 20 hours ago • verified publisher icon baidu

Hive Support

Open oclarms opened this issue 7 years ago • 2 comments

Read/write InputFormat/OutputFormat, SerDe from/to Hive Metastore. Read/write data from/to Hive Table or Partition.

oclarms avatar Nov 28 '17 03:11 oclarms

Do you have any plans or thoughts to support Hive IO formats?

If we are running on Spark, we can delegate the read path to Spark. However we have to implement all the Hive IO formats in cpp when running on local or writing data to Hive, which would be a quite complex project.

Another way would be delegating our work to hcatalog, that looks like a good direction.

advancedxy avatar Nov 28 '17 06:11 advancedxy

Yes, hcatalog is a good idea. But we need to test whether hcatalog supports the parquert file. Sometimes we want to specify the data storage format.

oclarms avatar Nov 29 '17 03:11 oclarms