web-llm icon indicating copy to clipboard operation
web-llm copied to clipboard

Published 20 hours ago • verified publisher icon mlc-ai

Can you write a tutorial for local deployment?

Open zsodur opened this issue 1 year ago • 37 comments

Can you write a tutorial for local deployment?

zsodur avatar Apr 15 '23 11:04 zsodur

+1

volkancirik avatar Apr 15 '23 12:04 volkancirik

Thank you for your suggestion. We will make a tutorial in the incoming weeks.

jinhongyii avatar Apr 15 '23 15:04 jinhongyii

can be run locally quickly like this:

git clone https://github.com/mlc-ai/web-llm

# switch branch
cd web-llm
git checkout -b gh-pages origin/gh-pages
cd docs

# start 
docker run --restart always  --name docker-web-llm -p 8060:80 -d -v "`pwd`:/usr/share/nginx/html" nginx
image

howie6879 avatar Apr 16 '23 16:04 howie6879

@howie6879 : thank you so much for the tutorial!

i don't use docker, but it works on my computer with serve ran into the doc folder, if I'm connected to the Internet.

I try to run it entirely locally, without an Internet connection at all, and I receive the following error message: Init error, TypeError: Cannot read properties of undefined (reading 'records')

Should I modify the "cacheUrl": "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/" line from the file "llm-chat-config.json"? If yes, what should I put for instead? I noticed that the files were downloaded to this folder: C:\Users\something\AppData\Local\Google\Chrome SxS\User Data\Default\Service Worker\CacheStorage\97999faa5d6050de6bffcb0ba574f502b068575b, but when I try to replace the current value with the foldername it didn't work offline.

Thank you so much for any hint | solution to how to make it work entirely offline.

zeritonius avatar Apr 16 '23 21:04 zeritonius

If the ask is just serve the website from the local machine, you can just serve base url as web-llm(note the baseurl argument below). Here is one example that can work with jekyll

git checkout gh-pages
cd docs
jekyll serve --host localhost --baseurl /web-llm --port 8888

Then you can open it on http://localhost:8888/web-llm

Other http server that have similar capabilities should also work

tqchen avatar Apr 16 '23 22:04 tqchen

@tqchen : thank you for the suggestion.

If I run jekyll serve --host localhost --baseurl /web-llm --port 8888 from inside the docs folder, I receive the same error if I'm offline [System Initalize] Init error, TypeError: Cannot read properties of undefined (reading 'records')

When I look into Google Chrome's Developer Fetch/XHR tab, it looks like it's trying to access https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/ndarray-cache.json which can't acess while I'm offline.

Is there any way to make it work entirely offline, by caching somehow the files it tries to access from HuggingFace?

zeritonius avatar Apr 16 '23 23:04 zeritonius

@zeritonius

I started using Nginx because Jekyll didn't work locally

You can install Docker or install nginx directly

howie6879 avatar Apr 17 '23 02:04 howie6879

@zeritonius If I understand well what you are trying to achieve:

cd docs

wget https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/ndarray-cache.json

for i in {0..162}
do
  wget "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_$i.bin"
done

then change the cache to: "cacheUrl": "http://localhost:8060/" seems a very hacky way of getting a local copy & being able to start offline - perhaps this is good enough for now?

eniradi avatar Apr 17 '23 14:04 eniradi

I don't know if it will help, but in my case I noticed it only works if the host is localhost (or 127.0.0.1).

I finally have it working by using jekyll in the docs directory with the command jekyll serve

kadogo avatar Apr 17 '23 16:04 kadogo

@eniradi Thanks to your indications, it now works the way I wished (locally and offline).

Locally = the code runs on my computer Offline = I don't need to be connected to Internet at any step. This is very important to me, because if the HuggingFace.co site is down at some point, I can still run the whole program in my browser (of course, after I downloaded all bin files, using your Bash script).

I prefer using the simplest tools possible and I'm glad to confirm that it works with serve as well, even when offline. serve is very small and it has an MIT license.

The solution was indeed making cacheUrl to point to a local folder. "cacheUrl": "http://localhost:3000/" (3000 for serve), instead of "cacheUrl": "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/" was the answer to my problem.

perhaps this is good enough for now?

It's perfect for me and way better than 24 hours ago. You made my day! Thank you so much!

@howie6879

I started using Nginx because Jekyll didn't work locally

serve worked fine all the way, but the cacheUrl's default value needed an Internet connection.

You can install Docker

I'd rather not, as I try to use open source tools whenever possible. From what I understand, Docker is nor open source, neither free for all.

install nginx directly

I use Windows and it seems that there is only a beta version for Windows : http://nginx.org/en/docs/windows.html

@kadogo

I finally have it working by using jekyll in the docs directory with the command jekyll serve

Does it work even if you're not connected to Internet? I'm not sure if Google Chrome has an "offline mode" as FireFox do. In my case, I removed the Internet cable to be 100% sure it works even offline.

Thanks to all whose suggestions made it possible for the script to work locally and offline.

So, to summarize all in one place, what worked for me:

git clone https://github.com/mlc-ai/web-llm
cd web-llm

git checkout -b gh-pages origin/gh-pages
cd docs

Edited the file "llm-chat-config.json"
(replacing "cacheUrl": "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/" with "cacheUrl": "http://localhost:3000/")

wget https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/ndarray-cache.json

for i in {0..162}
do
  wget "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_$i.bin"
done

serve

zeritonius avatar Apr 17 '23 17:04 zeritonius

Hello @zeritonius

I will let you know that, but I think I will still need the download of the cache like you have explained and serve looks much better thank Jekyll in my opinion because it can be installed without being related to the system.

My previous problem was that I was running the podman (or Jekyll) on my laptop and I couldn't make it work from my big computer. But running it on localhost made it work.

kadogo avatar Apr 17 '23 17:04 kadogo

When I do precisely what @zeritonius outlined the chat demo box reports that it's unable to initialize the WebGPU device:

This browser env do not support WebGPU Find an error initializing the WebGPU device Error: This browser env do not support WebGPU Init error, Error: Find an error initializing WebGPU: Error: This browser env do not support WebGPU

However, the chat box on the page at https://mlc.ai/web-llm/ works fine in the same Canary browser, and does not throw this error. I have checked that WebGPU is enabled, checked experimental flags, and all of that checks out.

My thought is that something about serve isn't giving the script what it needs, but I don't see why there would be a difference. I did have to modify the paths in index.html because they included the web-llm directory in their paths, which is above where /docs/ is and therefore not seen by serve.

What else could be causing the script to error like that? Something else with the wrong path?

(Chrome Canary, Win10) llm_chat.js:421 Error: This browser env do not support WebGPU llm_chat.js:547 Error: Find an error initializing WebGPU: Error: This browser env do not support WebGPU

intoempty avatar Apr 17 '23 23:04 intoempty

Hello @intoempty

I give it a try with python for serving static files because my system has some troubles with npm. I didn't alter the cacheUrl, but I guess that it would work too.

git clone https://github.com/mlc-ai/web-llm
cd web-llm

git checkout -b gh-pages origin/gh-pages
cd docs

# I use http://127.0.0.1:4000/ but I think only the localhost is mandatory
python3 -m http.server --bind 127.0.0.1 4000

I sometimes get the following error but then I just disable Unsafe WebGPU in chrome://flags/ and activate again and relaunch and it works (thanks to @tdockx). I didn't find the real issue yet.

Generate error, OperationError: The operation failed for an operation-specific reason

If I use something else than localhost I get an error similar to what you have

This browser env do not support WebGPU
Find an error initializing the WebGPU device Error: This browser env do not support WebGPU
Init error, Error: Find an error initializing WebGPU: Error: This browser env do not support WebGPU

kadogo avatar Apr 18 '23 00:04 kadogo

@kadogo

I didn't alter the cacheUrl, but I guess that it would work too.

Does it work even if you are not connected to the Internet? I believe your configuration works locally, but not offline. Of course, not everyone wants to also work offline - I was just curious if it works offline too in your case.

@intoempty

Could you, please, try this page and tell me if you see a big red triangle or an error message? https://webgpu.github.io/webgpu-samples/samples/helloTriangle

Although improbable, is it possible that you tried the local site in regular Chrome browser and the https://mlc.ai/web-llm/ in the Google Chrome Canary? What is the exact url you tried for the local site, please? I'm asking this because sometimes some files are open with the default browser, and I assume that Canary is not the default one.

I did have to modify the paths in index.html because they included the web-llm directory in their paths, which is above where /docs/ is and therefore not seen by serve.

You're right, but as far as I noticed, it only affects some logo from the very top of the page, way above the chat zone.

zeritonius avatar Apr 18 '23 00:04 zeritonius

@zeritonius

I just did the test and I confirmed, with my configuration it doesn't work offline (I guess because of the cacheUrl).

But I'm thinking the error about "This browser env do not support WebGPU" is related to the localhost. If I use the same setup, but I bind it on my local IP instead of 127.0.0.1 I'm getting the error.

kadogo avatar Apr 18 '23 00:04 kadogo

@kadogo I had altered cacheUrl, and updated the JSON as @zeritonius had indicated.

"wasmUrl": "/dist/vicuna-7b/vicuna-7b_webgpu.wasm", "cacheUrl": "http://172.24.3.245:3000/cache/",

(this happens to be the localnet IP through which it's possible to access WSL2 on Win10; I had it as localhost before but changed in case that was the issue.)

The logs of serve show that the app is getting everything it wants.

@zeritonius I see the red triangle. It's certain that I am running both apps in Canary. There has to be some path / other reason why the script does not think it's running in WebGPU environment.

intoempty avatar Apr 18 '23 00:04 intoempty

The download urls was changed. I use https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_0.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_1.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_2.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_3.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_4.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_5.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_6.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_7.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_8.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_9.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_10.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_11.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_12.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_13.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_14.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_15.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_16.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_17.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_18.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_19.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_20.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_21.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_22.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_23.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_24.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_25.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_26.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_27.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_28.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_29.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_30.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_31.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_32.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_33.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_34.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_35.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_36.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_37.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_38.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_39.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_40.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_41.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_42.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_43.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_44.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_45.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_46.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_47.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_48.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_49.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_50.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_51.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_52.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_53.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_54.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_55.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_56.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_57.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_58.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_59.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_60.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_61.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_62.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_63.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_64.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_65.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_66.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_67.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_68.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_69.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_70.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_71.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_72.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_73.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_74.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_75.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_76.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_77.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_78.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_79.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_80.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_81.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_82.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_83.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_84.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_85.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_86.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_87.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_88.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_89.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_90.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_91.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_92.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_93.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_94.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_95.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_96.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_97.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_98.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_99.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_100.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_101.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_102.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_103.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_104.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_105.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_106.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_107.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_108.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_109.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_110.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_111.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_112.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_113.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_114.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_115.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_116.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_117.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_118.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_119.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_120.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_121.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_122.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_123.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_124.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_125.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_126.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_127.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_128.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_129.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_130.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_131.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_132.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_133.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_134.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_135.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_136.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_137.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_138.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_139.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_140.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_141.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_142.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_143.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_144.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_145.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_146.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_147.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_148.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_149.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_150.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_151.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_152.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_153.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_154.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_155.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_156.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_157.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_158.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_159.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_160.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_161.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_162.bin

@eniradi 感谢您的指示，它现在可以照亮我希望的方式工作（本地和离线）。

本地=代码离线离线 离线= =我离线离线我我需要在在步骤任何连接连接到互联网互联网互联网互联网互联网。。。。这这这这这对对对对对对我来说来说来说来说来说来说来说来说非常非常离线离线离线离线离线离线离线离线离线运行整个程序（当然，在我下载所有bin文件之后，使用你的Bash脚本）。

我更喜欢尽管可能使用最简单的工具，我非常兴奋地认为它也可以与服务一起使用，即使在离线时也是如此。服务非常小，它有麻省理工学院可能。

解决方案确实使用 cacheUrl 指向本地文件夹。"cacheUrl": "http://localhost:3000/" （服务 3000），而不是"cacheUrl": "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/"我的问题的答案。

也许这就够了？

这对我来说非常完美，而且比24小时前好很多了。你让我今天一天都感觉很好！太感谢了！

@howie6879

我开始使用 Nginx 是因为 Jekyll 不能在本地工作

serve 一直运行良好，但 cacheUrl 的默认值需要 Internet 连接。

你可以安装 Docker

我宁愿不这样做，因为我尽管可能尝试使用开源工具。据我了解，Docker既不是开源的，也不是免费的。

直接安装nginx

我用的是Windows，好像只有Windows测试版：http://nginx.org/en/docs/windows.html

@kadogo

我最终通过使用 jekyll serve 命令在 docs 目录中使用 jekyll 使用其工作

即使即使没有到连接到到到连接也能能工作？不不不google chrome chrome浏览器chr浏览器浏览器是否是否是否是否是否是否是否一样一样一样一样一样一样一样一样一样离线离线离线离线离线。。。。我我也能正常工作。

感谢所有建议使脚本能在本地区和离线工作的人。

因此，总而言之，对我有用的是：

git clone https://github.com/mlc-ai/web-llm
cd web-llm

git checkout -b gh-pages origin/gh-pages
cd docs

Edited the file "llm-chat-config.json"
(replacing "cacheUrl": "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/" with "cacheUrl": "http://localhost:3000/")

wget https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/ndarray-cache.json

for i in {0..162}
do
  wget "https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_$i.bin"
done

serve

The download urls was changed. I use https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_0.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_1.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_2.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_3.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_4.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_5.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_6.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_7.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_8.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_9.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_10.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_11.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_12.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_13.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_14.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_15.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_16.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_17.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_18.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_19.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_20.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_21.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_22.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_23.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_24.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_25.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_26.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_27.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_28.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_29.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_30.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_31.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_32.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_33.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_34.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_35.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_36.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_37.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_38.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_39.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_40.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_41.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_42.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_43.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_44.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_45.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_46.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_47.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_48.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_49.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_50.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_51.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_52.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_53.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_54.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_55.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_56.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_57.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_58.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_59.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_60.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_61.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_62.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_63.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_64.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_65.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_66.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_67.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_68.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_69.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_70.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_71.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_72.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_73.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_74.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_75.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_76.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_77.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_78.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_79.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_80.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_81.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_82.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_83.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_84.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_85.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_86.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_87.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_88.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_89.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_90.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_91.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_92.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_93.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_94.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_95.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_96.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_97.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_98.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_99.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_100.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_101.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_102.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_103.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_104.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_105.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_106.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_107.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_108.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_109.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_110.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_111.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_112.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_113.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_114.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_115.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_116.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_117.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_118.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_119.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_120.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_121.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_122.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_123.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_124.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_125.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_126.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_127.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_128.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_129.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_130.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_131.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_132.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_133.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_134.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_135.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_136.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_137.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_138.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_139.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_140.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_141.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_142.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_143.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_144.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_145.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_146.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_147.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_148.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_149.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_150.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_151.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_152.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_153.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_154.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_155.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_156.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_157.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_158.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_159.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_160.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_161.bin https://huggingface.co/mlc-ai/web-lm/resolve/main/vicuna-0b/params_shard_162.bin

tmzncty avatar Apr 18 '23 05:04 tmzncty

The script snippet to download the shards worked fine for me. However, I noticed that visiting the local path to /dist/vicuna-7b/vicuna-7b_webgpu.wasm in Canary directly prompts a security warning before allowing that type of file to be downloaded in the browser. Could a local security restriction like this be blocking the script? (But if that were true, should also block when running off the demo site.

What else would cause WebGPU to fail to be initialized from the POV of this script?

intoempty avatar Apr 18 '23 14:04 intoempty

@intoempty

I noticed that visiting the local path to /dist/vicuna-7b/vicuna-7b_webgpu.wasm in Canary directly

I'm not sure what you mean by "directly", so I tried both: I opened the file from http://localhost:3000/vicuna-7b/vicuna-7b_webgpu.wasm I opened the file from "C:\Users\someone\Desktop\web-llm\docs\dist\vicuna-7b" folder

prompts a security warning before allowing that type of file to be downloaded in the browser.

In both cases, it opened it without any warning.

Here's the output if I click on "Downloads" in Canary: vicuna-7b_webgpu (1).wasm http://localhost:3000/dist/vicuna-7b/vicuna-7b_webgpu.wasm Show in folder

vicuna-7b_webgpu.wasm file:///C:/Users/someone/Desktop/web-llm/docs/dist/vicuna-7b/vicuna-7b_webgpu.wasm Show in folder

I have Version 114.0.5715.6 (Official Build) canary-dcheck (64-bit)

this happens to be the localnet IP through which it's possible to access WSL2 on Win10;

By the way, I don't use WSL, just Windows 10 (Home Edition) 64 bit.

I have checked that WebGPU is enabled, checked experimental flags, and all of that checks out.

How did you do that? That is, what was the address in the URL bar when you did that? Or was it in the Developer Tools section?

Have you also checked WebAssembly Debugging? There is another setting, called "local overrides for response headers" - maybe it's worth checking/unchecking (and restarting Canary after each such change, just in case) - maybe even a CTRL + SHIFT + R, as sometimes the served page is from cache.

I had it as localhost before but changed in case that was the issue.

I read this somewhere: Google Chrome 68 is here and as promised, the Chrome Browser is now marking HTTP as “not secure” (it's from 2018)

Would it be possible there is a setting in Canary that forbids http, and allows only https? As serve's URL is http, not https, but the official site is https. Firefox has such an option, but I'm not sure if Canary does and if it has, where it can be set.

In Developer Tools -- Settings -- Workspace, there is an "folder exclude" pattern. Have you tried deleting it to see if anything changes? This seems to apply to your computer only, and might prevent the script working locally but allowing it to run from the official site. /node_modules/|/bower_components/|/\.devtools|/\.git/|/\.sass-cache/|/\.hg/|/\.idea/|/\.svn/|/\.cache/|/\.project/|/Thumbs.db$|/ehthumbs.db$|/Desktop.ini$|/\$RECYCLE.BIN/

Inside Canary's Developer Tools, there is an WASM tab. When you receive "This browser env do not support WebGPU", could you, please, tell me what js script was running and at what line it stopped when that error appeared?

zeritonius avatar Apr 18 '23 16:04 zeritonius

@zeritonius thank you for the detailed questions and replies.

  • I did not delete the folder exclude option because none of the folders corresponded with the installation of web-llm
  • The error message in the developer console:


llm_chat.js:393 /dist/vicuna-7b/vicuna-7b_webgpu.wasm wasmURL
llm_chat.js:394 http://172.24.3.245:3000/cache/ cacheURL
llm_chat.js:422 Error: This browser env do not support WebGPU
    at #asyncInitTVM (llm_chat.js:418:15)
    at async LLMChatInstance.asyncInit (llm_chat.js:444:5)
    at async LLMChatInstance.generate (llm_chat.js:545:7)
    at async tvmjsGlobalEnv.asyncOnGenerate (llm_chat.js:607:3)
llm_chat.js:548 Error: Find an error initializing WebGPU: Error: This browser env do not support WebGPU
    at #asyncInitTVM (llm_chat.js:424:13)
    at async LLMChatInstance.asyncInit (llm_chat.js:444:5)
    at async LLMChatInstance.generate (llm_chat.js:545:7)
    at async tvmjsGlobalEnv.asyncOnGenerate (llm_chat.js:607:3)

The first two lines are log entries I added to the script so I could verify that it was getting the right locations

	console.log(wasmUrl + " wasmURL");
	console.log(cacheUrl + " cacheURL");

in case being unable to access those files (the .wasm and the various .bin files I have downloaded) was causing the WEbGPU to fail to init.

intoempty avatar Apr 20 '23 12:04 intoempty

jekyll serve --host localhost --baseurl /web-llm --port 8888

That worked for me. Thank you!

Bortus-AI avatar Apr 21 '23 18:04 Bortus-AI

Hi, all. We just add a local deployment instruction in the README. Welcome to check it out.

jinhongyii avatar Apr 22 '23 04:04 jinhongyii

How to migrate from 7b-0 to 7b-1 using (zeritonius) method? can anyone give me a hint? from docs folder.

rexzhang360 avatar Apr 23 '23 03:04 rexzhang360

@rexzhang360 try replacing all "vicuna-7b" to "vicuna-7b-v1"

jinhongyii avatar Apr 23 '23 04:04 jinhongyii

Where to find the vicuna-7b-v1 folder? I try to follow the new version of local deployment, but it fails at different stages. I could not find the vicuna-7b-v1 folder which includes the tokenizer and wasm files.

rexzhang360 avatar Apr 23 '23 05:04 rexzhang360

Could you describe the stage where you fails in details? If I understand your problem correctly, you are saying that you don't have path/to/vicuna-7b-v1 on your machine. The ways to get it is to follow the instructions here https://github.com/lm-sys/FastChat#vicuna-weights. the output directory is what you need.

jinhongyii avatar Apr 23 '23 05:04 jinhongyii

no, What I mean is the vicuna-7b-v1 is not under any newly deploy folder (which is failed up to the last final step ,deploy to local site)

rexzhang360 avatar Apr 23 '23 05:04 rexzhang360

What I want to be able upgrade from old deployment (where I use serve as webserver under docs folder , under docs there is dist folder include the vicuna-7b only)

rexzhang360 avatar Apr 23 '23 05:04 rexzhang360

After running all the steps, you will have a directory site/dist/vicuna-7b-v1

jinhongyii avatar Apr 23 '23 05:04 jinhongyii

it failed somewhere during the prep_dep.sh can not export file vicuan-7b-v1, udner dist folder only has two files, llm_chat.css llm_chat.js

rexzhang360 avatar Apr 23 '23 05:04 rexzhang360