llama icon indicating copy to clipboard operation
llama copied to clipboard

Published 20 hours ago • verified publisher icon ypeleg

GPU hardware for inference

Open dcolumbus1492 opened this issue 1 year ago • 2 comments

Can you please specify/estimate the hardware required to run each model? It will help me choose the correct EC2 instance. Many thanks!

dcolumbus1492 avatar Mar 08 '23 12:03 dcolumbus1492

for inference, it's like a 26GB VRAM card. for training, when i use 48GB, unfortunately, OOM.... still trying~ if you find proper devices, please tell me.

kriskrisliu avatar Mar 13 '23 15:03 kriskrisliu

@kriskrisliu According to what I read, training 7B model requires at least 4 x 80GB GPUs.

htcml avatar Apr 01 '23 17:04 htcml