[Usage]: Cleaning Up Stale/Abnormal Data in Metadata Server

[SpecterCipher]

Describe your usage question

Dear Maintainers and Community,

I'm currently experiencing an issue with the metadata server in moonccake where stale or abnormal data appears to be accumulating over time. The situation occurs when mooncake clients encounter repeated abnormal terminations and recoveries, causing continuous growth of data in the metadata server without proper cleanup of expired/invalid entries. I haven't been able to find a clear method to safely identify and remove the abnormal/stale data without potentially affecting active clients. Could you kindly provide guidance on:

1. What would be the recommended approach to clean up stale metadata?
2. Are there any built-in cleanup mechanisms I might have missed?

Thank you for your time and consideration.

Before submitting a new issue...

• [x] Make sure you already searched for relevant issues and read the documentation

[xiaguan]

I think we could consider storing the client's metadata in the Mooncake Master using client heartbeats. We might need to implement a MooncakeStoreMetaPlugin in the Transfer Engine.

[ykwd]

I think we could consider storing the client's metadata in the Mooncake Master using client heartbeats. We might need to implement a MooncakeStoreMetaPlugin in the Transfer Engine.

There is a built-in HttpMetadataServer in master, maybe we could improve the usability of meta service based on it.