RANGER-4932:Handling Specified key was too long; max key length error

[basapuram-kumar]

What changes were proposed in this pull request?

Proposed changes are mentioned at RANGER-4932

How was this patch tested?

tested this patch on my local cluster during the migration of ranger from 2.3.0 to 2.5.0, and with proposed changes, able to fix this issue.

[prabhjyotsingh]

LGTM!

@spolavarpau1 / @princeap173 can you please help review.

[bhavikpatel9977]

Changing the DEFAULT CHARSET for just one table might cause issue if the DEFAULT CHARSET is set to something different.

[basapuram-kumar]

Hello @bhavikpatel9977 , I have following approaches to discuss here.

1. Define the DEFAULT CHARSET=utf8 has been successful.
2. Reduce the size of indexed columns either with
   KEYx_trx_log_v2_trx_id (trx_id(512))) or
   KEY x_trx_log_v2_trx_id (trx_id(255)) also seems to work..

Could you please advise whether one of these approaches would be optimal, or if there are any better alternatives we should consider? Thank you.

[basapuram-kumar]

Hi team, I'm just checking in on the status of my pull request. Would you be able to review it soon? Would appreciate any feedback or suggestions on this PR. Thanks! cc: @bhavikpatel9977 , @spolavarpau1 , @princeap173 , @kumaab , @prabhjyotsingh

[basapuram-kumar]

Hi team, I'm just checking in on the status of my pull request. Would you be able to review it soon? Would appreciate any feedback or suggestions on this PR. Thanks

[vikaskr22]

Hi @basapuram-kumar , Thanks for the work ! Today I tried to understand this, and sharing my input on this. I see merit in Bhavik's comment, changing the CHARSET at table level may create many problems. Charset changes in just one table may break JOIN operation or may result wrong output. Sometimes, we set the Charset at Connection level in code , like we set to read using UTF-8, there it would break.

My input is, it should be defined at schema/DB level ( may be using DB server configs), means same charset for entire Ranger.

Now coming to Latin1, it will limit the characters that Ranger supports or will support in future.

On the second approach, where you suggested to reduce the length of trx_id(255), it may work. But if intention is to keep exact transactionId in the index, then it would be an issue.

Few questions: What is the MySql version where you got this error ? What is default "storage_engine"? It should be 'InnoDb'.

I am also exploring this and will add my input here.

[basapuram-kumar]

Hello @vikaskr22 , Thanks for the details explaination on the pros and cons of the suggested settings.

yes, absolutely agreed on the pointers.

For the asked info,

What is the MySql version where you got this error ? What is default "storage_engine"? It should be 'InnoDb'.

I am using MySQL-8.0.36, and the detault storage_engine is InnoDB

mysql > show create table x_trx_log_v2;

x_trx_log_v2 | CREATE TABLE x_trx_log_v2 ( id bigint NOT NULL AUTO_INCREMENT, create_time datetime DEFAULT NULL, added_by_id bigint DEFAULT NULL, class_type int NOT NULL DEFAULT '0', object_id bigint DEFAULT NULL, parent_object_id bigint DEFAULT NULL, parent_object_class_type int NOT NULL DEFAULT '0', parent_object_name varchar(1024) DEFAULT NULL, object_name varchar(1024) DEFAULT NULL, change_info mediumtext, trx_id varchar(1024) DEFAULT NULL, action varchar(255) DEFAULT NULL, sess_id varchar(512) DEFAULT NULL, req_id varchar(30) DEFAULT NULL, sess_type varchar(30) DEFAULT NULL, PRIMARY KEY (id), KEY x_trx_log_v2_FK_added_by_id (added_by_id), KEY x_trx_log_v2_cr_time (create_time), KEY x_trx_log_v2_trx_id (trx_id) ) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC |