Persistent Parquet Metadata Cache for Object Storage

[Selfeer]

Describe the new feature

Currently, we lose all cached data when the server restarts, which impacts performance and requires recalculating or refetching data. To improve reliability and efficiency, we need a mechanism to persist the cache across restarts.

Expected Behavior

• Cached data should be retained even after a server restart.
• The system should be able to reload the cache from a backup storage (e.g., disk, or external caching service).

Possible Solutions

The usage of Redis might be beneficial here, as when a new node starts or an existing one restarts, it can already access the cached metadata from Redis.