H2O-GPT on AMD GPUs (ROCm)

[rohitnanda1443]

Hi, How can we run H20-GPT on AMD-GPUs using the AMD ROCm libraries.

One can easily run an inference server on Ollama using ROCm thereby H2O-GPT needs to use this Ollama server for inferencing.

Problem: H2o-GPT install fails as it keeps finding CUDA during install. Some guidance here on editing the install script for ROCm would be helpful,

Method:

1. LLM runs on an inference server using ROCm
2. H2o-GPT sends LLM requests to the inference server