dalai
dalai copied to clipboard
cocktailpeanut
Requests use 7B model when only 65B model is installed
Installed Dalai via "npx dalai llama 65B" to only download the 65B parameter model. This successfully downloaded and quantized. Once this completed I ran "npx dalai serve" to run the web page.
The web page successfully runs, and from the "models" drop down only 65B is available.
When I enter a request and hit autocomplete nothing happens. Looking at the server output for the query, it lists "model: '7B' " for every request I make. At the very bottom there is another line that states "models: ['65B']" but every request still appears to be attempting to use the 7B model. I believe this may be why they're failing for me
I am garbage at node.js but if it looks like the query is incorrect. Do you know how to change the query?
Server running on http://localhost:3000/
query: { method: 'installed' } modelsPath /home/xdoestech/llama.cpp/models { modelFolders: [ '13B', '30B', '65B', '7B' ] } exists 65B query: { seed: -1, threads: 4, n_predict: 1000, model: '7B', THIS SHOULD BE 65B top_k: 40, top_p: 0.9, temp: 0.8, repeat_last_n: 64, repeat_penalty: 1.3, models: [ '65B' ], prompt: 'the fuck' }
ok so this runs in the terminal but does not run in the webUI... I know am missing something basic here
const Dalai = require('dalai');
new Dalai().request({ model: "65B", prompt: "The following is a conversation between a boy and a girl:", }, (token) => { process.stdout.write(token) })
new Dalai().serve(3000)
I am garbage at node.js but if it looks like the query is incorrect. Do you know how to change the query?
Server running on http://localhost:3000/
query: { method: 'installed' } modelsPath /home/xdoestech/llama.cpp/models { modelFolders: [ '13B', '30B', '65B', '7B' ] } exists 65B query: { seed: -1, threads: 4, n_predict: 1000, model: '7B', THIS SHOULD BE 65B top_k: 40, top_p: 0.9, temp: 0.8, repeat_last_n: 64, repeat_penalty: 1.3, models: [ '65B' ], prompt: 'the fuck' }
I can confirm that this is what my queries look like as well. A quick fix is to simply also install the 7B model. I can confirm that after I installed the 7B model, and select 65B from the web ui the queries then come through successfully.
Seems like there is some form of bug where if there is only one model installed it assumes that the model is 7B.
Unfortunately I haven't had time to dig through the code and figure out why this is happening or how to fix the query
i wonder if making a symlink to the 30B/65B/etc folder would work instead? haven't tried it myself but as long as it's not expecting an exact match of the model's files it might be worth a shot as a temporary workaround at least
The issue is the website defaults to 7B and only updates the value when you have a change event on the drop down. Since there's only one item in the drop down a change event will never fire and never update the model.
No the code I posted above works just the webui does not work with it. There are instructions to make the webui but I do not have time to learn socket.io until this weekend. The api works try the code above.
The issue is the website defaults to
7Band only updates the value when you have a change event on the drop down. Since there's only one item in the drop down a change event will never fire and never update the model.
Do you know how to access the code to change this? Ive looked throught the github and dont see anything (I have no idea what I am doing however). I have no storage for another model at this point
As a workaround just change the default here https://github.com/cocktailpeanut/dalai/blob/main/bin/web/views/index.ejs#L115
