[Feature Request] add eviction API's to VM's module and type caches

[sblackshear]

Problem

The Move VM keeps a cache of the modules it has previously deserialized, verified, and linked (and similar for types and scripts). However, there are no API's for removing entries from these caches, nor is there any eviction policy. This means that an instance of the VM is kept alive and continues encountering new modules, these caches will grow without bound and eventually cause memory issues.

This was not an issue in Diem because the set of modules was fixed and VM instances were ephemeral (i.e., the system starts a new VM instance for each block). But other uses of Move may want to use the VM in a different way--e.g., we are considering a system where a server keeps an instance of the VM alive indefinitely, and we expect the server to encounter many different modules.

Proposed solution

I think we should start with exposing API's that:

1. Return the size of all three caches in bytes
2. Clear all caches

This is rather crude, but these API's will at least allow adapters to prevent the caches from growing beyond N bytes.

In time, I can imagine VM config's that allow adapter's specifying various eviction policies (e.g., a LFU cache that drops the long tail of infrequently used modules seems like a sensible one), but I'd rather move forward with something simple for now and use experience/data to help us design something fancier.

[vgao1996]

Yeah, this has long been a known issue and hopefully we'll have the bandwidth to tack it in the following days.

I think we should start with exposing API's that:

• Return the size of all three caches in bytes
• Clear all caches

I think these are reasonable short-term solutions that will get you unblocked while we explore the option of lifting the code cache out of the Move VM. @runtian-zhou did a very similar thing previously, so I don't think we'll be worried about a few more patches.

In time, I can imagine VM config's that allow adapter's specifying various eviction policies (e.g., a LFU cache that drops the long tail of infrequently used modules seems like a sensible one), but I'd rather move forward with something simple for now and use experience/data to help us design something fancier.

Yeah, I think we'll definitely need to consider something like an LFU cache in the future. Hopefully allowing the adapter to take ownership of the code cache will make this easier to implement.