private-gpt
private-gpt copied to clipboard
zylon-ai
Mac m2 query slow
Note: if you'd like to ask a question or open a discussion, head over to the Discussions section and post it there.
Describe the bug and how to reproduce it I use a 8GB ggml model to ingest 611 MB epub files to gen 2.3GB db. And I query a question, it took 40 minutes to show the result. And I found the python process read 1.2TB data from disk.
Expected behavior I have 24 GB memory in my mac mini, the model and db size is 10GB, then the process could hold all data to memory rather than read data from disk so many time. And the cost time is too long.
Environment (please complete the following information):
- OS / hardware: [e.g. macOS 12.6 / M2 mac mini 24GB memory]
- Python version [e.g. 3.10.6]
Having the same issue with an even smaller database. My db is 1.2 MB in size, my source documents are all PDFs that total less then 40MB in size. Disk read from python is 1.31TB.
My env file:
PERSIST_DIRECTORY=db MODEL_TYPE=LlamaCpp MODEL_PATH=models/ggml-vic13b-q5_1.bin EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2 MODEL_N_CTX=1000
OS/Hardware: macOS 13.4.1 / M2 Pro 16GB RAM
Python: 3.11.1
