How to disable CUDA when installing through docker (run CPU only)

[AndreiSva]

I'm trying to install through docker, but I don't have an nvidia gpu. How can I configure the .env file to install the webui on a computer without CUDA support?

[Morriz]

@oobabooga would this be an easy answer for you?

[oobabooga]

I honestly don't know. Maybe @loeken can help

[loeken]

not tested but i think if you remove the section:

# deploy:
#   resources:
#     reservations:
#       devices:
#         - driver: nvidia
#           device_ids: ['0']
#           capabilities: [gpu]

from the docker compose.yml it should work

[baonq-me]

If the server doesn't have any GPU, you must take one more step:

• Open file ./modules/GPTQ_loader.py
• Go to line 79
• Change the line to this model.load_state_dict(torch.load(checkpoint, map_location=torch.device('cpu')), strict=False)

[Morriz]

tnx, but this should be made configurable then

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.