amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[ARCTIC-459][FLINK] Fix the missing of the required fields after project pushdown

Open zstraw opened this issue 3 years ago • 0 comments

FIX #459

Why are the changes needed?

Project pushdown may lead to the missing of primary key, partition key. ShuffleHelper may lose the required info, and produce NPE.

This hotfix tends to read the required fields appended after the projected fields in the order of 'projected fields', 'primary keys', 'partition keys'. And send the rowData with additional fields to downstream.

Brief change log

  • The flink 1.12 and 1.14, 1.15 versions are affected.

How was this patch tested?

  • [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [ ] Add screenshots for manual tests if appropriate

  • [X] Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

zstraw avatar Oct 12 '22 11:10 zstraw