connectors icon indicating copy to clipboard operation
connectors copied to clipboard

add ability to disable schema validation when creating external Hive tables using the delta-hive connector

Open jwbreedlove opened this issue 2 years ago • 2 comments

We are experiencing very slow performance when creating external hive tables using the DeltaStorageHandler through the Hive/beeline shells. The slow performance doesn't always seem consistent but it mainly occurs with tables that are significant in size, ie number or rows or columns. I am speculating that this slow performance is due to the schema validation that is taking places when using the Delta Hive connector.

Would it be possible to add a property to disable schema validation upon external table creation? In the event a tables schema does change we have a process to automate dropping and recreating the table to account for the schema changes.

In some cases we are seeing create table statements run for upwards of an hour.

jwbreedlove avatar Apr 28 '22 17:04 jwbreedlove

@zsxwing what do you think about this?

scottsand-db avatar May 04 '22 17:05 scottsand-db

I think a better solution is just don't require users to provide the schema (#285)

zsxwing avatar May 04 '22 17:05 zsxwing

This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See delta-io/connectors#556 for details.

vkorukanti avatar Jul 11 '23 17:07 vkorukanti