amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[ARCTIC-108][Flink] Support projection push down to arctic source reader

Open SteNicholas opened this issue 3 years ago • 1 comments

Fix #108.

ArcticDynamicSource should implement the SupportsProjectionPushDown interface to accelerate data querying.

The brief change log

  • ArcticDynamicSource and KafkaDynamicSource implement the SupportsProjectionPushDown interface to support supportsNestedProjection and applyProjection methods.

SteNicholas avatar Aug 15 '22 03:08 SteNicholas

@SteNicholas Thanks for your work. LogDataJsonDeserialization will deserialize the byte[] data to RowData by Iceberg's Schema. You can find related code here. I think we should also change the schema when the projection is pushed down. What do you think?

YesOrNo828 avatar Aug 17 '22 03:08 YesOrNo828

@YesOrNo828, thanks for your suggestion. I have addressed the above comments and updated the schema in LogDynamicSource for deserialization. PTAL.

SteNicholas avatar Aug 19 '22 06:08 SteNicholas