santacoder-finetuning icon indicating copy to clipboard operation
santacoder-finetuning copied to clipboard

Published 20 hours ago • verified publisher icon loubnabnl

Why is the inference speed using fp16 or bf16 similar to fp32?

Open lionday opened this issue 1 year ago • 1 comments

Is there any specific configuration method? model = AutoModelForCausalLM.from_pretrained(checkpoint,trust_remote_code=True,torch_dtype=torch.float16)

lionday avatar Jul 07 '23 08:07 lionday