onediff icon indicating copy to clipboard operation
onediff copied to clipboard

Published 20 hours ago • verified publisher icon siliconflow

CPU Model Offloading

Open abi98213 opened this issue 1 year ago • 7 comments

Does oneflow support model offloading like pipe.to('cpu') while all the graphs are being loaded ?

abi98213 avatar May 04 '23 20:05 abi98213

If anyone can please answer this @doombeaker

abi98213 avatar May 06 '23 01:05 abi98213

offloading like pipe.to('cpu') is not currently supported.

We are working on a feature like this( CPU offload), but it will not be ready to use quickly.

Please tell us why you want this feature, and has any tool provided this feature?

strint avatar May 06 '23 01:05 strint

I need this feature, since I have 8 Gigs GPU and I cannot load multiple models within the GPU. I have to move models to cpu to load any other model. This thing is easily doable with diffusers like this : pipe.enable_model_cpu_offload(). So I was wondering if that is possible with oneflow Thanks

abi98213 avatar May 08 '23 03:05 abi98213

This thing is easily doable with diffusers like this : pipe.enable_model_cpu_offload().

This feature is not ready currently. We are thinking about it.

strint avatar May 09 '23 03:05 strint

How much time this can take ?

abi98213 avatar May 09 '23 10:05 abi98213

@strint If you can please answer

abi98213 avatar May 27 '23 17:05 abi98213

I'll leave a comment as I need this mechanism too

shabri-arrahim avatar Aug 20 '23 11:08 shabri-arrahim

This issue is too old to follow and will be closed. Feel free to reopen it and continue.

strint avatar Dec 19 '23 20:12 strint