[Bug]: Some history records in database are not dropped as expected

[zhoujinsong]

What happened?

Some records created before 30 days existed in table_optimizing_process/task_runtime/optimizing_task_quota database tables with self-optimizing.runtime-data-keep-days configuration is set to 30.

Affects Versions

master

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

It seems the dropped tables left some records in the database will not be deleted any more.

Relevant log output

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[ihadoop]

Is it means when table is dropped， it should clean all optimizing data immediately.

In addition, we should remove table history recorders in database which cannot find table name in any catalog 。 Could you confirm is it right? @zhoujinsong

[zhoujinsong]

Is it means when table is dropped， it should clean all optimizing data immediately. In addition, we should remove table history recorders in database which cannot find table name in any catalog

Both yes for these two questions.

[zhoujinsong]

The current clean method for the optimizing histories is scheduled separately for each table. I believe this design can be simplified, and this issue can be resolved simultaneously:

• Schedule the cleaning process for all tables together.
• Create a daily partition for history tables and change the clean SQL to drop partition.

[ihadoop]

plz assign this issue to me

[Jzjsnow]

@ihadoop Hi, I would like to know how this issue is going, we are experiencing similar problems.

[zhoujinsong]

It seems not to be fixed for now. Do you have time to address and fix it? @ihadoop

[ihadoop]

I will submit pr next weekend

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'