connectors
connectors copied to clipboard
add ability to disable schema validation when creating external Hive tables using the delta-hive connector
We are experiencing very slow performance when creating external hive tables using the DeltaStorageHandler through the Hive/beeline shells. The slow performance doesn't always seem consistent but it mainly occurs with tables that are significant in size, ie number or rows or columns. I am speculating that this slow performance is due to the schema validation that is taking places when using the Delta Hive connector.
Would it be possible to add a property to disable schema validation upon external table creation? In the event a tables schema does change we have a process to automate dropping and recreating the table to account for the schema changes.
In some cases we are seeing create table statements run for upwards of an hour.
@zsxwing what do you think about this?
I think a better solution is just don't require users to provide the schema (#285)
This repo has been deprecated and the code is moved under connectors
module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See delta-io/connectors#556 for details.