Query execution hangs for 07_custom.py

[aschroede]

Hi there, I was trying to get the 07_custom.py program to run with the facebook/opt-iml-1.3b model and I can see it loads the cache correctly and I put in enough print statements to see that it also was able to get the LLMPredictor, create the service context, and load the index from disk. However when it tries to call execute_query the program seemingly hangs. I can see my RAM usage spike for an extended period of time but no matter how long I wait (20 minutes?) I don't get a response from the model. Note that I am running with an AMD GPU so when creating the pipeline I removed the CUDA device specification because as far as I can tell CUDA Is not supported with AMD GPUs. Do I need a more powerful computer or CUDA to run this?

Here are my specifications:

OS: Windows 11 Processor AMD Ryzen 7 5800H with Radeon Graphics 3.20 GHz Installed RAM 16.0 GB (13.9 GB usable) Device ID XXXXXXXXXXXXXX Product ID 00342-20715-34612-AAOEM System type 64-bit operating system, x64-based processor GPU 0: AMD radeon RX 6600M GPU1: AMD Radeon(TM) Graphcis Pen and touch Pen support

Thanks for your help!