[FEATURE REQUEST] Archival support for metadata table of Record level indexing

[bibhu107]

Hi Community,

I am seeking guidance on implementing archival policies for the Metadata table in Hudi's record-level indexing (RLI). We are not concerned with retaining data older than three years, as we move that data from standard S3 storage to Glacier. Additionally, we would like to archive the metadata record keys provided by Hudi's record-level indexing to keep storage costs under control. I wanted to inquire if there are any plans to introduce TTL or Archival support for the Hudi metadata table's record keys. Is there any plan to bring this feature.

Thanks

cc - @codope

[danny0405]

I wanted to inquire if there are any plans to introduce TTL or Archival support for the Hudi metadata table's record keys.

Record level ttl is a very tipical request for regular K-V storage, so as to the RLI.

[bibhu107]

@danny0405 Are you planning to create a JIRA ticket for same? We started using RLI but we will need support in creating TTL policies for RLI

[danny0405]

Not really, you can create a JIRA to trace it.