amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[ARCTIC-910]Ignore committing error to HMS when committing Hive format table transaction

Open hameizi opened this issue 2 years ago • 3 comments

…saction

Why are the changes needed?

Close #910 .

Brief change log

  • Ignore committing error to HMS when committing Hive format table transaction

How was this patch tested?

  • [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [ ] Add screenshots for manual tests if appropriate

  • [ ] Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not documented)

hameizi avatar Dec 28 '23 06:12 hameizi

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (5eb153f) 32.64% compared to head (92c2f16) 32.59%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2482      +/-   ##
============================================
- Coverage     32.64%   32.59%   -0.05%     
+ Complexity     4473     4469       -4     
============================================
  Files           598      598              
  Lines         50276    50280       +4     
  Branches       6676     6676              
============================================
- Hits          16411    16389      -22     
- Misses        32550    32587      +37     
+ Partials       1315     1304      -11     
Flag Coverage Δ
core 30.75% <100.00%> (+<0.01%) :arrow_up:
trino 50.38% <ø> (-0.58%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 28 '23 06:12 codecov[bot]

Can you share the exception stack for the scenario addressed by this PR? I'm not sure when this issue occurs. @hameizi

wangtaohz avatar Dec 29 '23 05:12 wangtaohz

I also noticed that it seems like the try-catch block was forgotten to be added here com.netease.arctic.hive.op.ReplaceHivePartitions#commit

  @Override
  public void commit() {
    if (!addFiles.isEmpty()) {
      List<DataFile> dataFiles =
          HiveCommitUtil.commitConsistentWriteFiles(this.addFiles, table.io(), table.spec());
      this.addFiles.clear();
      this.addFiles.addAll(dataFiles);
      this.addFiles.forEach(delegate::addFile);

      commitTimestamp = (int) (System.currentTimeMillis() / 1000);
      if (table.spec().isUnpartitioned()) {
        generateUnpartitionTableLocation();
      } else {
        applyHivePartitions();
      }

      delegate.commit();
      commitPartitionProperties();
      if (!insideTransaction) {
        transaction.commitTransaction();
      }

      // here the try-catch is missing
      if (table.spec().isUnpartitioned()) {
        commitUnPartitionedTable();
      } else {
        commitPartitionedTable();
      }
    }
  }

wangtaohz avatar Dec 29 '23 05:12 wangtaohz

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions[bot] avatar Aug 29 '24 00:08 github-actions[bot]