localGPT

localGPT copied to clipboard

This isn't ideal:

@click.command()
@click.option('--device_type', default='cuda', help='device to run on, select gpu, cpu or mps')
def main(device_type, ):
    # load the instructorEmbeddings
    if device_type in ['cpu', 'CPU']:
        device='cpu'
    elif device_type in ['mps', 'MPS']:
        device='mps'
    else:
        device='cuda''

Instead it would be better to do the following:

@click.command()
@click.option('--device_type', default=None, help='device to run on, select cpu, cuda, hip, or mps')
def main(device_type):
    # load the instructorEmbeddings
    device_type = device_type.lower() if device_type else 'cpu'

Here, we supply the list of devices supported by pytorch in the help text instead.

You can get the list by passing an invalid device name to pytorch device method.

This way, it falls back to the CPU as a default if anything goes wrong and you don't have to create branches that check for every possible type.

@teleprint-me I agree with you, this could be a better solution. However, my thought process is if someone has a GPU on their system, then localGPT should use that by default without the user setting it up. If there is no GPU or anything goes wrong, we default to CPU. GPU utilization is one of the main reasons for it to be different that privateGPT. How would you implement that? Thanks.

That's why it's a command line parameter and we include it in the help message.

A sane default would be cuda.

@click.command()
@click.option('--device_type', default='cuda', help='device to run on, select cpu, cuda, hip, or mps')
def main(device_type):
    # omitting branch logic because all of the logic is already handled for us

Let's say I use AMD and I have all of the pre-requisites installed... then I should be able to do the following:

Ξ ~ → python             
Python 3.11.3 (main, Apr  5 2023, 15:52:25) [GCC 12.2.1 20230201] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> 

Even though I'm running AMD and using ROCm instead of Nvidia and using CUDA, I still get the same result.

Ξ ~ → lspci | grep -i vga
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]... (rest of device info)

If I make a mistake with naming the pytorch device, it let's me know.

>>> torch.device('hips')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: hips
>>> 

The reason all of this is relevant is because most of the underlying code already handles it (sentence-transformers, langchain, etc). All the user needs to do is specify their device. There's no real supporting rationale for attempting to predict the users device. This will only lead to frustration for contributors and users otherwise.

Makes sense and I like the approach. Can you please create a PR with updates to the Readme? Thanks

It honestly doesn't make any sense. A lot of the code is wrapped and "cuda" is literally hard coded everywhere. I can get rocm to work with hip on its own, but the moment I use any python pacakages besides pytorch is the moment it automatically pulls in a whole bunch of nvidia deps that mask my system install (this is a much deeper issue rooted at all sorts of levels in pretty much any source I've seen so far for python). It's been... interesting... trying to get this stuff to work on an AMD GPU.