private-gpt icon indicating copy to clipboard operation
private-gpt copied to clipboard
verified publisher icon zylon-ai

performance issue privateGPT

Open mehrdad2000 opened this issue 2 years ago • 6 comments

Hi I've consider 40 core cpu/ 40GB mem/ssd disk for privateGPT but still slowing when it will answer!

Any idea or trick to improve performance?

FYI: set n_threads=40 in this file privateGPT.py to use all cpu cores.

Thanks

mehrdad2000 avatar May 22 '23 12:05 mehrdad2000

gpu support has been just cracked so it should be soon merged with main branch

2blackbar avatar May 22 '23 15:05 2blackbar

@2blackbar how it work? If it merge still required gpu or it will work with cpu with better performance?

mehrdad2000 avatar May 22 '23 15:05 mehrdad2000

GPU support will be awesome for inference, the ingestion is super fast as it is on CPU. Looking forward to trying it.

thanks Darrin

darrinh avatar May 23 '23 07:05 darrinh

@darrinh my issue about slow answer not slow ingest!

mehrdad2000 avatar May 28 '23 01:05 mehrdad2000

Yes, I have the same issue, was saying that ingest works fine as it is on CPU.

darrinh avatar May 28 '23 07:05 darrinh

@2blackbar any update? Does it affect speed of answer or just improve ingest time?

mehrdad2000 avatar May 28 '23 14:05 mehrdad2000