Request: Estimated RAM Requirement

Open brian-at-pieces opened this issue 7 months ago • 1 comments

It would be helpful if the foundry model info <model name> command included a rough estimate of the RAM required to run each model so that we could gate some models to avoid issues with users trying to run ones that are too large for their machine. If that's not possible, it would at least be nice to know the data type for the model weights so we can calculate a rough estimate of RAM at runtime. Maybe just a "Quantization" column is needed? Thanks!

AB#66984

Apr 29 '25 16:04 brian-at-pieces

Unrelated but it would also be nice for info to report the model context size

Apr 29 '25 18:04 brian-at-pieces