infer on multi 4090 cards

[hyhzl]

i only have 4 x 4090 cards, but i can not run internvl-v1-5, which the program will run model on the same card, resulting in OOM. how could i run model on cards rather than on only one card @czczup

[NiYueLiuFeng]

you can sea example of multi gpu for inference from https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5.

# Otherwise, you need to set device_map='auto' to use multiple GPUs for inference.
model = AutoModel.from_pretrained(
     path,
     torch_dtype=torch.bfloat16,
     low_cpu_mem_usage=True,
     trust_remote_code=True,
     device_map='auto').eval()