Loading models from disk

[bakkot]

First, this is extremely cool.

Second: I note that there are an absolute ton of models derived from Stable Diffusion around. Depending on your device, you might be able to fit a few of them in the cache TVM uses, but it requires duplicating the entire model, which is a waste of disk space if you have a local copy. (This matters a lot when you have dozens of models i.e. hundreds of GBs' worth.) And if it gets evicted from the cache, or try to run it in another browser or an incognito window or whatever, you have to download it again.

It would be nice if you could keep your model files on disk and load them into memory without making a copy (either using the file system API or drag-and-drop). Is that something that could happen? Poking around a bit, it seems like maybe that would require a change to TVM, but I don't know for sure.

(Someday I'd like to get to a place where whole fancy UIs for stable diffusion, including all of the computation, can be hosted on Github Pages, ideally multiple such UIs. And for multiple UIs to be practical, they'll need to be able to load models from disk, since they can't share the cache.)

[MasterJH5574]

Hi @bakkot, thanks for your great suggestion. Loading from local files is not hard. In the future we can provide such an option for people to choose, on “whether to download and use browser cache or to load the model files from disk.” We can also provide an option for people to download the model files to disk for future runs instead of always using browser cache. Thanks again for the suggestion!

[DustinBrett]

It would also be good if we could pass in a full 2GB+ model file, without the need to break it up.