aibrix icon indicating copy to clipboard operation
aibrix copied to clipboard
verified publisher icon vllm-project

StreamLoader can persist models to disk in bypass

Open brosoul opened this issue 1 year ago • 1 comments

🚀 Feature Description and Motivation

StreamLoader can persist models to disk in bypass. This way can avoid the network transmission overhead of loading the model from the current machine for the second time.

Use Case

No response

Proposed Solution

No response

brosoul avatar Nov 18 '24 09:11 brosoul

Due to limited time, I will untrack this issue from v0.2.0.

Jeffwan avatar Dec 16 '24 18:12 Jeffwan