[BUG/Help] 支持low_cpu_mem_usage加载

[digits122]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

当在AutoModel.from_pretrained添加参数low_cpu_mem_usage=True之后，会报错误 RuntimeError: Only Tensors of floating point and complex dtype can require gradients

Expected Behavior

使用low_cpu_mem_usage=True时，一般是想将模型加载到gpu，并且让此过程中最小的cpu内存消耗，以满足在小内存、大显存机器上的加载。无论是否有此特性，不应该加上之后报此错误。

Steps To Reproduce

AutoModel.from_pretrained增加参数low_cpu_mem_usage=True

Environment

- OS:ubuntu22.04
- Python:3.10.6
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response