Step 1

🟢 Nvidia GPU ➜ Install CUDA 12.1+

🔴 AMD GPU - (AMD GPU acceleration support coming in future releases).

Step 2

Download the ZIP file for the latest "release," extract its contents, navigate to the "src" folder to run the commands below.

• NOTE: If you clone this repository you WILL NOT get the latest release. Instead, you will development versions of this program which may or may not be stable.

Step 3

Navigate to the src folder, open a command prompt, and create a virtual environment:

python -m venv .

Step 4

Activate the virtual environment:

.\Scripts\activate

Step 5

Run setup:

python setup.py

Optional Step 6

Run this command if you want to doublecheck that you installed the Pytorch and gpu-acceleration software correctly:

python check_gpu.py

Linux users must use Release v3.5.2 until I can update the codebase due to recent major changes. Download the ZIP file for that release and follow the instructions in the readme.md.

MacOS users must use Release v3.5.2 until I can update the codebase due to recent major changes. Download the ZIP file for that release and follow the instructions in the readme.md.

Activate Virtual Environment

• Once you install the program you've already created a virtual environment, so you just need to activate it each time you want to restart it. Remember to run the appropriate command to do so (based on your platform) within the src folder.

Start the Program

python gui.py

Only systems with an Nvidia GPU will display gpu power, usage, and VRAM metrics.

🔥Important🔥

• Read the User Guide before sending me questions.

Download Vector Model

• In the Models Tab tab, choose the embedding model you want to download. The User Guide Tab contains information about the various models.

Create a Vector Database

• In the Create Database tab, click Choose Files and select one or more files to add. This can be repeated as many times as you wish.
  • Supported documents are: .pdf, .docx, .epub, .txt, .html, .enex, .eml, .msg, .csv, .xls, .xlsx, .rtf, .odt.
• If you selected any image files, I highly recommend that you adjust the vision model settings within the Settings tab and testing the settings while processing a single image, which can easily be done within the Tools tab.
  • Supported images are: .png, .jpg, .jpeg, .bmp, .gif, .tif, .tiff
• 🔥 To add audio files you must transcribe one or more audio files from the Tools Tab. The transcriptions will be saved and added when you create the vector database.
  • Supported audio extensions include, but are not limited to: .mp3, .wav, .m4a, .ogg, .wma
• Click Choose Model.
• Click 🔥ONCE🔥 on the directory containing the vector model you want to use and then click "OK."
• In the input box, enter a name for the database you want to create (accepts numbers, lowercase letters, and the characters "-" or "_").
• In the Settings Tab, set the chunk size, chunk overlap, and the device you want to use. More information is in the User Guide.
• Click the Create Vector Database button.
  • 🔥 MAKE SURE to wait until the command prompt states that the database has been successfully created before proceeding.

Delete a Database

• In the Manage Databases tab, select a database from the pulldown menu and click Delete Database.

  The ability to delete one or more specific files is coming soon.

Query a Database (No LM Studio)

• In the Query Database tab, select the database you want to use from the pulldown menu.
• Enter your question by typing it or using the Record Question button.
• Check the chunks only checkbox.
• Click Submit Question, which will return chunks relevant to your query.
  • In the Settings tab, you can change multiple settings regarding querying the database. More information can be found in the User Guide.

Query a Database with a Response From LM Studio

This program can get relevant chunks from the vector database and forwarding them - along with your question - to LM Studio for an answer!

• Perform the above steps regarding entering a question and choosing settings, but make sure that Chunks Only is 🔥UNCHECKED🔥.
• Start LM Studio and go to the Server tab on the left.
• Load a model. I've personally tested the following models as good:
  • Marx_3B_V3
  • Mistral_7B_Instruct_v0_2
  • Neural_Chat_7b_v3_3
  • Llama_2_13b_chat_hf
  • SOLAR_10_7B_Instruct_v1_0
• Turn Apply Prompt Formatting to "OFF."
• On the right side within Prompt Format, make sure that all of the following settings are blank:
  • System Message Prefix
  • System Message Suffix
  • User Message Prefix
  • User Message Suffix
• At the top, load a model within LM Studio.
• On the right, adjust the GPU Offload setting to your liking.
• Within my program, go to the Settings tab, select the appropriate prompt format for the model loaded in LM Studio, click Update Settings.
• In LM Studio, click Start Server.
• In the Query Database tab, click Submit Question.