gpu_poor icon indicating copy to clipboard operation
gpu_poor copied to clipboard

Published 20 hours ago • verified publisher icon RahulSChand

why batch size does not effect to memory usage in inference mode

Open LaniakeaS opened this issue 1 year ago • 0 comments

Different batch size doesn't seem to affect GPU memory usage when set in INFERENCE MODE? This doesn't seem to make sense. Is that normal？

LaniakeaS avatar Dec 13 '23 02:12 LaniakeaS