Out of physical memory when loading model

[frankang]

trafficstars

Hi, I got the following error when loading llama-7B model (ported to huggingface), my server has 256GB RAM on board. Is there any option to reduce the RAM consumption? Thanks. fatal : Memory allocation failure or ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 5 (pid: 32708) of binary: /home/user/miniconda3/envs/pytorch1.12/bin/python