lorax icon indicating copy to clipboard operation
lorax copied to clipboard

Skip the download process from `lorax-launcher` if model weights already on local disk

Open tgaddair opened this issue 1 year ago • 1 comments
trafficstars

Download process here will be essentially a no-op if the model weights are already present, but this can add several seconds of latency to startup.

We can make a quick check from within the lorax-launcher to see if the model weights exist and if so, skip this call entirely.

tgaddair avatar Jan 12 '24 18:01 tgaddair

If I may add some suggestion, please take care to make HUGGING_FACE_HUB_TOKEN variable optional, because even if I intend to serve local model only, LoRAX demands this variable ... Btw I use the model that is on HF gated repo.

micholeodon avatar Jul 11 '24 07:07 micholeodon