hive icon indicating copy to clipboard operation
hive copied to clipboard

HIVE-28300: Fix AlterTableConcatenate when using Hive-Tez

Open ngsg opened this issue 1 year ago • 1 comments

What changes were proposed in this pull request?

Modify AlterTableConcatenateOperation to refer to MergeFileWork.getInputPaths().

Why are the changes needed?

When using a List Bucketing table, FileMergeOperator expects only 1 subdirectory as its input. To satisfy this requirement, AlterTableConcatenateOperation changes input paths by calling MergeFileWork.resolveConcatenateMerge(). But Hive-Tez does not properly propagate the changed input paths to SplitGenerator, which causes the problem

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

Using list_bucket_dml_8.q. This PR changes the test driver for list_bucket_dml_8.q from TestCliDriver to TestMiniLlapLocalCliDriver, as all the other list_bucket_dml_*.q are already run by TestMiniLlapLocalCliDriver. Also MR specific config keys are replaced by corresponding Tez specific keys. I changed the file format of table from RCFile to ORC because ORC is more popular than RCFile and the issue comes from their common class: AbstractFileMergeOperator.

ngsg avatar Jun 04 '24 10:06 ngsg

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Jun 04 '24 11:06 sonarqubecloud[bot]