Travis Johnson

Thanks again for opening up a very detailed issue! In general, I don't think that the controller is hardened for a use-case where many SRs are created dynamically with ISVCs....

To provide some additional context: the error log shows that there have been 8 attempts at retrying the request within ModelMesh. Since the models are allocated to instances dynamically and...

Hmm, yeah hard to say if there aren't any apparent resource constraints 🤔. I think we'll need to find some more details as to what is causing the failures. How...

> There are 2500 model, each ~170MB (425 GB), so definitely don't fit into the available memory (22 GB). 😮 That's awesome! > It might be the estimated size of...

> It might be that our load is very uniform, so that the LRU cache doesn't really help that much. The typical load is simply call each of the 2500...

Hello, Thanks for including details in your issue report! If the empty dir's space was being exceeded, I would expect the pod to crash with an error making that clear...

I think there may be a little confusion. @andreaTP what you are referring to is versioning of objects with a `versionId` which is a particular feature of S3 implementations supports,...

I agree strongly with this statement 😄 : > I understand that implementing this feature may require careful consideration and planning, as well as potential changes to the underlying architecture

Just adding a bit more context: The current configuration that specifies the `idExtractionPath` for `rpcConfigs` is JSON content hard-coded into [cluster_config.go](https://github.com/kserve/modelmesh-serving/blob/91b610bd1bcd8d5264dcd696db4e2b65730f7468/controllers/modelmesh/cluster_config.go#L37-L57). The implementation for this issue would allow that list...