amoro
amoro copied to clipboard
[Feature]: support filter function for self-optimizing.filter
Description
Use case: in some produce case, we only need to optimize the lastest n partitions. the old partition will be override by offline data sync. for example offline data repair。
how to use: 1、for hidden partition, support self-optimizing.filter = 'max(days(ts),5)' 2、for partition, support self-optimizing.filter = 'max(pt,5)'
Use case/motivation
Parse the functions in the expression and return the specified Iceberg filter expression.
Describe the solution
Parse the functions in the expression and return the specified Iceberg filter expression.
Subtasks
No response
Related issues
No response
Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct