seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Paimon]support projection for paimon source

Open TaoZex opened this issue 1 year ago • 5 comments

Purpose of this pull request

support projection for paimon source

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

TaoZex avatar Feb 07 '24 06:02 TaoZex

PTAL @Hisoka-X @EricJoy2048

TaoZex avatar Apr 11 '24 06:04 TaoZex

cc @dailai

Hisoka-X avatar Apr 11 '24 07:04 Hisoka-X

Please support projection by field name.

dailai avatar Apr 11 '24 07:04 dailai

Please support projection by field name.

Fixed. Thanks.

TaoZex avatar Apr 11 '24 12:04 TaoZex

LGTM

dailai avatar Apr 12 '24 00:04 dailai

Hi @TaoZex , could you solve the conflict?

Hisoka-X avatar Jun 04 '24 12:06 Hisoka-X

Hi @TaoZex , could you solve the conflict?

Of course.

TaoZex avatar Jun 04 '24 12:06 TaoZex

For https://github.com/apache/seatunnel/pull/6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

Carl-Zhou-CN avatar Jun 06 '24 03:06 Carl-Zhou-CN

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, https://github.com/apache/seatunnel/pull/6887 can be further through SQL for the projection.

TaoZex avatar Jun 06 '24 03:06 TaoZex

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, #6887 can be further through SQL for the projection.

@TaoZex I think 'read_columns' can be overlaid in 'query sql'. Can we transform this pr based on sql？

Carl-Zhou-CN avatar Jun 06 '24 03:06 Carl-Zhou-CN

For #6887 There also seems to be an intention to cover the projection, at which point we will have two projection parameters,Do we need to keep both? @Hisoka-X @TaoZex @dailai

This pr implements the projection for paimon. After the pr was merged, #6887 can be further through SQL for the projection.

@TaoZex I think 'read_columns' can be overlaid in 'query sql'. Can we transform this pr based on sql？

I had a discussion with @dailai on wechat. Currently his pr only supports select * sql queries. I will wait for his pr to merge, and modify this pr code to select id, name sql query (Assuming multiple fields, but only two fields are queried by projection) cc @hailin0 @Hisoka-X

TaoZex avatar Jun 07 '24 01:06 TaoZex

this pr https://github.com/apache/seatunnel/pull/6887 is merged @TaoZex

hailin0 avatar Jun 12 '24 15:06 hailin0

this pr #6887 is merged @TaoZex

I have changed the code. PTAL. @Hisoka-X @hailin0

TaoZex avatar Jun 13 '24 10:06 TaoZex

cc @dailai

hailin0 avatar Jun 13 '24 15:06 hailin0

LGTM @hailin0

Thanks for review. Done.

TaoZex avatar Jun 14 '24 11:06 TaoZex