amoro
amoro copied to clipboard
[ARCTIC-459][FLINK] Fix the missing of the required fields after project pushdown
FIX #459
Why are the changes needed?
Project pushdown may lead to the missing of primary key, partition key. ShuffleHelper may lose the required info, and produce NPE.
This hotfix tends to read the required fields appended after the projected fields in the order of 'projected fields', 'primary keys', 'partition keys'. And send the rowData with additional fields to downstream.
Brief change log
- The flink 1.12 and 1.14, 1.15 versions are affected.
How was this patch tested?
-
[X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
-
[ ] Add screenshots for manual tests if appropriate
-
[X] Run test locally before making a pull request
Documentation
- Does this pull request introduces a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)