amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[Feature]: data-retention, add support for partition column type Date

Open vanphuoc3012 opened this issue 6 months ago • 1 comments

Description

Currently, data expire feature only support timestamp/timestampz/long type and string type field in date format

data-expire.field NULL Field used to determine data expiration, supporting timestamp/timestampz/long type and string type field in date format

It would be nice if support partition column type Date.

Use case/motivation

amoro-ams 2025-06-20 10:19:10,468 WARN [async-data-expiring-executor-0] [org.apache.amoro.server.table.TableConfigurations] [] - Table(iceberg-hive-catalog.db.my_table) field(etl_date) type(DATE) is not supported for data exp iration, please use the following types: LONG, TIMESTAMP, STRING

Describe the solution

No response

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

vanphuoc3012 avatar Jun 20 '25 10:06 vanphuoc3012

@vanphuoc3012 thanks for reaching out for this feature. Would you like to try implementing it? It may be necessary to update the logic in TableConfiguration#validateExpirationField and IcedbergTableMaintainer#getDataExpression

klion26 avatar Jun 23 '25 07:06 klion26

@vanphuoc3012 thanks for reaching out for this feature. Would you like to try implementing it? It may be necessary to update the logic in TableConfiguration#validateExpirationField and IcedbergTableMaintainer#getDataExpression感谢您联系我们获取此功能。您想尝试实现它吗?可能需要更新 TableConfiguration#validateExpirationFieldIcedbergTableMaintainer#getDataExpression 中的逻辑。

@klion26 Hello, I want to try it, can you assign the task to me?

lsyulong avatar Jul 03 '25 08:07 lsyulong

Hi @vanphuoc3012 , can you please assign this to me.

MadhuriRathod30 avatar Jul 03 '25 13:07 MadhuriRathod30

Hi @vanphuoc3012 , can you please assign this to me.

Hi @klion26 , I think I dont have perrmission to assign this issue to people 😁

vanphuoc3012 avatar Jul 05 '25 03:07 vanphuoc3012

Thanks for the interesting about contributing to this issue @lsyulong @MadhuriRathod30 and sorry for the late reply. I'll assign this to @lsyulong this time, as he/she commented first. There are some good first issue if we want to try, and we can also file a pr directly when there is no assignee and pr next time.

Please ping me if there is something I can help with.

klion26 avatar Jul 07 '25 08:07 klion26

Thanks for the interesting about contributing to this issue @lsyulong @MadhuriRathod30 and sorry for the late reply. I'll assign this to @lsyulong this time, as he/she commented first. There are some good first issue if we want to try, and we can also file a pr directly when there is no assignee and pr next time.

Please ping me if there is something I can help with.

@klion26 I have modified the relevant code logic. If you have time, please help review the code to see if there are any other issues. I will make further adjustments. Thank you very much

lsyulong avatar Jul 11 '25 08:07 lsyulong