Speedway1

For locally hosted LLMs this feature is essential, because it allows for different machines to provide each type of LLM which enables for more GPUs to be used preloaded with...

@rasodu Thank you for the above. However, for me, same version of Ubuntu and kernel as you, when I made those changes, I got this: root@TH-AI2:~# apt install amdgpu-dkms Reading...

OK in addition to the steps from @rasodu above, you need to: apt update && apt upgrade -y amdgpu-install --usecase=graphics,opencl --no-32 --no-dkms Not sure about the last 2 parameters just...

OK I needed to completely reinstall the ROCM drivers, but resetting to version 6.1 per @rasodu 's suggestion worked. I am now properly inferencing across multiple Radion 7900XTX GPUs.

Given that we have the solution (reset to earlier version of the AMD drivers), probably this issue can be marked as resolved?

> @Speedway1 if you think putting drivers to an earlier level is a solution do that but for real operations that is not a solution that is maybe a workaround....

> If we have a proper trace why this happens we could open a bug with AMD, i think this works though with normal Tensor on python, I'm not convinced...

Actually there may be something deeper here. It may be that there's an assumption (based on using powerful models like GPT-4o) that the text analysis model can write good JSON...

Yes, agreed, I was doing further tests and came to the same conclusion. I was using "pitbull" https://huggingface.co/fblgit/UNA-ThePitbull-21.4B-v2 But then I swapped over to command-r which works much better but...

> Hey @Speedway1 this is definitely an issue with output inconsistency in LLMs. GPT Researcher is fully tested using GPT but would love the communities support in improving it for...